2024-03-16T20:13:12.431 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T20:13:12.431 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148 --owner scheduled_leonidus@teuthology --verbose --name leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/orig.config.yaml 2024-03-16T20:13:12.485 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T20:13:12.581 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148 branch: wip-lusov-qdb-squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} email: null first_in_suite: false job_id: '7602148' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb-squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - SLOW_OPS - slow request sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb-squid sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6918 sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb-squid suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhYT+mWvk67WkaGjYH/wdHRHq9iI0Lby+JQlD1OW15yt6mlEqOEsmJ7JtVqbcRCV2Ki0ncnWm6QHsmJgaCfT3o= smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9MPcoe3E/e07wdBbvHbRKpaqOB46smt9Kkhd1sk0urdTVn27eReBzD3rZBqcQDu8Le07vvygTCbP/01dXPN1Q= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE5c01L4cWx83yFfc9ux633eti6d9hT9EtU1qPXoT6baIaeUPmiZeHDgBUwfuoR2FFDgLvaBgtfCtm3xDvGUMOI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_11:44:52 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T20:13:12.581 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa; will attempt to use it 2024-03-16T20:13:12.582 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/qa/tasks 2024-03-16T20:13:12.582 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T20:13:12.584 INFO:teuthology.task.internal:Checking packages... 2024-03-16T20:13:12.606 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b' 2024-03-16T20:13:12.606 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T20:13:12.606 INFO:teuthology.packaging:ref: None 2024-03-16T20:13:12.607 INFO:teuthology.packaging:tag: None 2024-03-16T20:13:12.607 INFO:teuthology.packaging:branch: wip-lusov-qdb-squid 2024-03-16T20:13:12.607 INFO:teuthology.packaging:sha1: aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:13:12.607 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb-squid 2024-03-16T20:13:12.827 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1681.gaba3c1d2 2024-03-16T20:13:12.829 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T20:13:12.835 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T20:13:12.835 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T20:13:12.865 INFO:teuthology.task.internal:Saving configuration 2024-03-16T20:13:12.884 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T20:13:12.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T20:13:12.922 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.086758', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhYT+mWvk67WkaGjYH/wdHRHq9iI0Lby+JQlD1OW15yt6mlEqOEsmJ7JtVqbcRCV2Ki0ncnWm6QHsmJgaCfT3o='} 2024-03-16T20:13:12.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.083363', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9MPcoe3E/e07wdBbvHbRKpaqOB46smt9Kkhd1sk0urdTVn27eReBzD3rZBqcQDu8Le07vvygTCbP/01dXPN1Q='} 2024-03-16T20:13:12.963 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.085199', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE5c01L4cWx83yFfc9ux633eti6d9hT9EtU1qPXoT6baIaeUPmiZeHDgBUwfuoR2FFDgLvaBgtfCtm3xDvGUMOI='} 2024-03-16T20:13:12.963 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T20:13:12.969 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-16T20:13:12.969 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-16T20:13:12.969 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-16T20:13:12.969 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T20:13:13.090 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbdb17ef0d0>, signals=[15]) 2024-03-16T20:13:13.091 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T20:13:13.097 INFO:teuthology.task.internal:Opening connections... 2024-03-16T20:13:13.097 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:13:13.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:13:13.205 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-03-16T20:13:13.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:13:13.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-16T20:13:13.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:13:13.398 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T20:13:13.404 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-16T20:13:13.423 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-16T20:13:13.423 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-16T20:13:13.480 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T20:13:13.482 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T20:13:13.483 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-16T20:13:13.504 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-03-16T20:13:13.522 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-03-16T20:13:13.522 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-03-16T20:13:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-03-16T20:13:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-03-16T20:13:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-03-16T20:13:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-03-16T20:13:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T20:13:13.580 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T20:13:13.580 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-03-16T20:13:13.604 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-16T20:13:13.623 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-16T20:13:13.623 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T20:13:13.680 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-16T20:13:13.681 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T20:13:13.681 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T20:13:13.681 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T20:13:13.681 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-16T20:13:13.704 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T20:13:13.715 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T20:13:13.740 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T20:13:13.740 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-16T20:13:13.743 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-03-16T20:13:13.745 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-16T20:13:13.761 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T20:13:13.766 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T20:13:13.766 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-16T20:13:13.800 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-03-16T20:13:13.803 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-16T20:13:13.905 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T20:13:13.981 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T20:13:13.995 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-16T20:13:13.995 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-16T20:13:13.996 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-16T20:13:13.996 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T20:13:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:13:14.013 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-16T20:13:14.071 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-361.el9.x86_64 2024-03-16T20:13:14.071 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.14.0-361.el9.x86_64 2024-03-16T20:13:14.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2024-03-16T20:13:16.382 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 9.7 MB/s | 6.7 MB 00:00 2024-03-16T20:13:17.891 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-16T20:13:23.040 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-16T20:13:24.974 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-16T20:13:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-16T20:13:30.676 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-03-16T20:14:37.145 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 319 kB/s | 21 MB 01:06 2024-03-16T20:14:42.397 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-16T20:14:44.626 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T20:14:44.627 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T20:14:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-16T20:14:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-16T20:14:44.724 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-16T20:14:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-16T20:14:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout:Install 4 Packages 2024-03-16T20:14:44.725 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:14:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 100 M 2024-03-16T20:14:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 123 M 2024-03-16T20:14:44.726 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-16T20:14:45.146 INFO:teuthology.orchestra.run.smithi023.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-03-16T20:14:45.713 INFO:teuthology.orchestra.run.smithi023.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-03-16T20:14:46.305 INFO:teuthology.orchestra.run.smithi023.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 29 MB/s | 33 MB 00:01 2024-03-16T20:14:46.647 INFO:teuthology.orchestra.run.smithi023.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-03-16T20:14:46.647 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-16T20:14:46.648 INFO:teuthology.orchestra.run.smithi023.stdout:Total 52 MB/s | 100 MB 00:01 2024-03-16T20:14:47.567 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-16T20:14:48.014 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-16T20:14:48.014 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-16T20:14:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-16T20:14:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-16T20:14:52.306 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-16T20:14:52.995 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T20:14:53.131 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T20:14:56.053 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T20:14:56.234 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T20:15:01.331 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T20:15:01.454 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:06.584 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:37.457 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:37.458 INFO:teuthology.orchestra.run.smithi023.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T20:15:37.458 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:15:37.543 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T20:15:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T20:15:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T20:15:38.454 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:15:38.455 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-16T20:15:38.992 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T20:15:39.143 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T20:15:39.143 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T20:15:39.143 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T20:15:39.143 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T20:15:39.143 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T20:15:39.144 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-16T20:15:39.144 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T20:15:39.144 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T20:15:39.269 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T20:15:39.269 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T20:15:39.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.06 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 68.el9 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Group : Unspecified 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 8264254 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-16T20:15:39.315 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : builder@centos.org 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-16T20:15:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:15:39.317 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T20:15:39.317 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T20:15:39.318 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 5.14.0-427.el9.x86_64 2024-03-16T20:15:39.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T20:15:39.937 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-16T20:15:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T20:15:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-16T20:15:41.380 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-16T20:15:41.396 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.sIk2cQzAHM 2024-03-16T20:15:41.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sIk2cQzAHM 2024-03-16T20:15:41.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.sIk2cQzAHM 2024-03-16T20:15:41.583 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.sIk2cQzAHM is 6KB 2024-03-16T20:15:41.631 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.sIk2cQzAHM 2024-03-16T20:15:41.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T20:15:41.717 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T20:15:41.718 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T20:15:41.718 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T20:15:41.718 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T20:15:41.720 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T20:15:41.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-16T20:16:11.860 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-16T20:16:11.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:16:11.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:16:30.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-16T20:16:39.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:16:39.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:16:42.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-16T20:16:54.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:16:54.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:16:57.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-16T20:17:12.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:17:12.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:17:15.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-16T20:17:33.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:17:33.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:18:05.281 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-16T20:18:05.658 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:18:05.659 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-16T20:18:05.659 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-16T20:18:05.674 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-427.el9.x86_64 2024-03-16T20:18:05.674 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T20:18:05.675 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T20:18:05.675 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T20:18:06.676 DEBUG:teuthology.parallel:result is None 2024-03-16T20:18:06.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T20:18:06.683 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T20:18:06.684 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T20:18:06.686 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T20:18:06.689 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T20:18:06.706 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T20:18:06.712 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T20:18:06.763 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T20:18:06.763 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T20:18:06.766 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T20:18:06.768 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T20:18:06.888 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T20:18:06.895 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T20:18:06.896 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T20:18:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:06.913 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T20:18:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:06.929 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T20:18:06.932 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T20:18:06.956 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T20:18:07.000 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.006 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.011 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.022 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.029 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.057 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T20:18:07.059 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T20:18:07.066 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T20:18:07.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T20:18:07.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T20:18:07.071 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T20:18:07.103 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T20:18:07.117 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T20:18:07.117 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T20:18:07.139 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T20:18:07.145 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T20:18:07.161 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.293 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.420 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:18:07.420 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T20:18:07.489 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.597 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.691 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:18:07.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T20:18:07.764 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.808 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T20:18:07.872 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.916 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T20:18:07.960 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:18:07.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T20:18:08.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-16T20:18:08.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-03-16T20:18:08.035 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-16T20:18:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T20:18:08.081 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T20:18:08.106 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T20:18:08.597 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T20:18:08.604 INFO:teuthology.task.internal:Starting timer... 2024-03-16T20:18:08.604 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T20:18:08.649 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T20:18:08.698 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-16T20:18:08.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-16T20:18:08.768 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-16T20:18:08.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-03-16T20:18:08.835 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-03-16T20:18:08.837 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-16T20:18:08.891 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-16T20:18:08.893 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T20:18:08.893 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-16T20:18:08.920 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-16T20:18:08.921 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-03-16T20:18:08.958 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-03-16T20:18:08.958 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-16T20:18:08.981 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-16T20:18:08.981 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-03-16T20:18:08.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T20:18:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:09.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T20:18:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:09.043 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T20:18:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T20:18:09.070 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T20:18:09.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-16T20:18:09.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-03-16T20:18:09.144 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-16T20:18:09.174 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T20:18:09.184 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T20:18:09.200 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T20:18:09.201 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi031.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T20:22:52.557 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-03-16T20:22:52.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:22:52.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:22:52.648 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-16T20:22:52.733 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-16T20:22:52.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-16T20:22:52.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:22:52.835 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-03-16T20:22:52.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-03-16T20:22:52.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-16T20:22:52.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T20:22:53.008 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-16T20:22:53.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-16T20:22:53.082 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T20:22:53.093 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T20:22:53.093 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T20:22:53.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T20:22:53.096 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T20:22:53.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T20:22:53.099 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T20:22:53.099 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T20:22:53.127 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T20:22:53.134 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T20:22:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T20:22:53.154 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T20:22:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T20:22:53.171 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T20:22:53.174 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-16T20:22:53.188 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-16T20:22:53.192 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-03-16T20:22:53.201 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-16T20:22:53.205 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T20:22:53.209 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-03-16T20:22:53.215 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-16T20:22:53.222 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T20:22:53.229 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T20:22:53.231 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T20:22:53.247 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T20:22:53.249 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T20:22:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.277 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.291 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-16T20:22:53.295 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T20:22:53.296 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-16T20:22:53.296 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.296 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.296 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.296 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.297 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.300 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T20:22:53.301 INFO:teuthology.run_tasks:Running task install... 2024-03-16T20:22:53.309 DEBUG:teuthology.task.install:project ceph 2024-03-16T20:22:53.310 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'}} 2024-03-16T20:22:53.310 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'} 2024-03-16T20:22:53.310 INFO:teuthology.task.install:Using flavor: default 2024-03-16T20:22:53.317 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-16T20:22:53.317 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-16T20:22:53.318 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'tag': None, 'wait_for_package': False} 2024-03-16T20:22:53.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:22:53.320 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'tag': None, 'wait_for_package': False} 2024-03-16T20:22:53.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:22:53.321 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'tag': None, 'wait_for_package': False} 2024-03-16T20:22:53.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:22:53.465 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/ 2024-03-16T20:22:53.466 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1681.gaba3c1d2 2024-03-16T20:22:53.469 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/ 2024-03-16T20:22:53.469 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1681.gaba3c1d2 2024-03-16T20:22:53.477 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/ 2024-03-16T20:22:53.477 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1681.gaba3c1d2 2024-03-16T20:22:53.586 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T20:22:53.586 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:22:53.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T20:22:53.618 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T20:22:53.618 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T20:22:53.641 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T20:22:53.641 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:22:53.641 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T20:22:53.646 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-16T20:22:53.646 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:22:53.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-16T20:22:53.669 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T20:22:53.670 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T20:22:53.673 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-16T20:22:53.673 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-16T20:22:53.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T20:22:53.738 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T20:22:53.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-16T20:22:53.777 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T20:22:53.806 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-03-16T20:22:53.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-03-16T20:22:53.817 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T20:22:53.818 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-16T20:22:53.883 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-03-16T20:22:53.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-03-16T20:22:53.885 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-16T20:22:53.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-16T20:22:54.054 INFO:teuthology.orchestra.run.smithi031.stdout:74 files removed 2024-03-16T20:22:54.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T20:22:54.120 INFO:teuthology.orchestra.run.smithi189.stdout:74 files removed 2024-03-16T20:22:54.132 INFO:teuthology.orchestra.run.smithi023.stdout:74 files removed 2024-03-16T20:22:54.147 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T20:22:54.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-16T20:22:54.784 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 211 kB/s | 83 kB 00:00 2024-03-16T20:22:54.835 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 209 kB/s | 83 kB 00:00 2024-03-16T20:22:54.846 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 229 kB/s | 83 kB 00:00 2024-03-16T20:22:55.235 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-16T20:22:55.309 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-16T20:22:55.363 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-16T20:22:55.710 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-16T20:22:55.752 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-16T20:22:55.797 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-16T20:22:56.246 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-16T20:22:56.516 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-16T20:22:57.005 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 5.5 MB/s | 6.7 MB 00:01 2024-03-16T20:22:57.713 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-16T20:22:57.975 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-16T20:22:58.397 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-16T20:22:58.570 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-16T20:22:58.791 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-16T20:22:59.221 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-16T20:23:01.130 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-16T20:23:01.358 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-16T20:23:05.385 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-16T20:23:05.389 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-16T20:23:06.794 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 15 kB 00:00 2024-03-16T20:23:07.006 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-16T20:23:08.577 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-16T20:23:09.703 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-03-16T20:23:13.716 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-16T20:23:14.787 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-16T20:23:16.120 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-16T20:23:16.281 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:16.282 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:16.297 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T20:23:16.298 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T20:23:16.299 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T20:23:16.300 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:16.302 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:16.305 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T20:23:16.306 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T20:23:16.392 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-03-16T20:23:16.419 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-16T20:23:16.419 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-03-16T20:23:16.419 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 6.6 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.3 M 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 887 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 145 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 1.5 M 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 138 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 1.8 M 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 7.4 M 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 49 k 2024-03-16T20:23:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 9.8 M 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 47 M 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 264 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 748 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 218 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 27 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 18 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 31 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 744 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 127 k 2024-03-16T20:23:16.421 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 161 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 321 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 298 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 99 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 85 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 171 k 2024-03-16T20:23:16.422 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T20:23:16.423 INFO:teuthology.orchestra.run.smithi031.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.4 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T20:23:16.424 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 20 M 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 31 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 2.3 M 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 248 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 4.7 M 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 17 M 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 15 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 25 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T20:23:16.425 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 163 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 498 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T20:23:16.426 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.1 M 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T20:23:16.427 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 45 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 131 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T20:23:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T20:23:16.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T20:23:16.430 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T20:23:16.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T20:23:16.432 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout:Install 150 Packages 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 24 Packages 2024-03-16T20:23:16.433 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:23:16.434 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 224 M 2024-03-16T20:23:16.434 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-03-16T20:23:17.118 INFO:teuthology.orchestra.run.smithi031.stdout:(1/174): ceph-19.0.0-1681.gaba3c1d2.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-16T20:23:17.418 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:17.419 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:17.434 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T20:23:17.435 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T20:23:17.436 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T20:23:17.437 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:17.440 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:17.443 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T20:23:17.444 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T20:23:17.529 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T20:23:17.542 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 6.6 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.3 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 887 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 145 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 1.5 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 138 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 1.8 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 7.4 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 49 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 9.8 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 47 M 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 264 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 748 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 218 k 2024-03-16T20:23:17.543 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 27 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 18 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 31 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 744 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 127 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 161 k 2024-03-16T20:23:17.544 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 321 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 298 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 99 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 85 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 171 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T20:23:17.545 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.4 M 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T20:23:17.546 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 20 M 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 31 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 2.3 M 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 248 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 4.7 M 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 17 M 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 15 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 25 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T20:23:17.547 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 163 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 498 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T20:23:17.548 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.1 M 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T20:23:17.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 45 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 131 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T20:23:17.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T20:23:17.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T20:23:17.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T20:23:17.553 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-16T20:23:17.554 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-16T20:23:17.555 INFO:teuthology.orchestra.run.smithi023.stdout:Install 150 Packages 2024-03-16T20:23:17.555 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 24 Packages 2024-03-16T20:23:17.555 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:23:17.555 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 224 M 2024-03-16T20:23:17.555 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-16T20:23:17.601 INFO:teuthology.orchestra.run.smithi031.stdout:(2/174): ceph-fuse-19.0.0-1681.gaba3c1d2.el9.x8 1.8 MB/s | 887 kB 00:00 2024-03-16T20:23:17.760 INFO:teuthology.orchestra.run.smithi031.stdout:(3/174): ceph-immutable-object-cache-19.0.0-168 920 kB/s | 145 kB 00:00 2024-03-16T20:23:18.135 INFO:teuthology.orchestra.run.smithi031.stdout:(4/174): ceph-common-19.0.0-1681.gaba3c1d2.el9. 17 MB/s | 20 MB 00:01 2024-03-16T20:23:18.252 INFO:teuthology.orchestra.run.smithi031.stdout:(5/174): ceph-mgr-19.0.0-1681.gaba3c1d2.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-16T20:23:18.380 INFO:teuthology.orchestra.run.smithi023.stdout:(1/174): ceph-19.0.0-1681.gaba3c1d2.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-16T20:23:18.577 INFO:teuthology.orchestra.run.smithi031.stdout:(6/174): ceph-mon-19.0.0-1681.gaba3c1d2.el9.x86 14 MB/s | 4.7 MB 00:00 2024-03-16T20:23:18.627 INFO:teuthology.orchestra.run.smithi031.stdout:(7/174): ceph-mds-19.0.0-1681.gaba3c1d2.el9.x86 2.7 MB/s | 2.3 MB 00:00 2024-03-16T20:23:18.728 INFO:teuthology.orchestra.run.smithi031.stdout:(8/174): ceph-base-19.0.0-1681.gaba3c1d2.el9.x8 2.9 MB/s | 5.3 MB 00:01 2024-03-16T20:23:18.811 INFO:teuthology.orchestra.run.smithi031.stdout:(9/174): ceph-selinux-19.0.0-1681.gaba3c1d2.el9 302 kB/s | 25 kB 00:00 2024-03-16T20:23:19.355 INFO:teuthology.orchestra.run.smithi023.stdout:(2/174): ceph-fuse-19.0.0-1681.gaba3c1d2.el9.x8 910 kB/s | 887 kB 00:00 2024-03-16T20:23:19.563 INFO:teuthology.orchestra.run.smithi023.stdout:(3/174): ceph-immutable-object-cache-19.0.0-168 698 kB/s | 145 kB 00:00 2024-03-16T20:23:19.878 INFO:teuthology.orchestra.run.smithi031.stdout:(10/174): ceph-osd-19.0.0-1681.gaba3c1d2.el9.x8 13 MB/s | 17 MB 00:01 2024-03-16T20:23:19.962 INFO:teuthology.orchestra.run.smithi031.stdout:(11/174): cephfs-mirror-19.0.0-1681.gaba3c1d2.e 2.6 MB/s | 218 kB 00:00 2024-03-16T20:23:20.037 INFO:teuthology.orchestra.run.smithi031.stdout:(12/174): cephfs-shell-19.0.0-1681.gaba3c1d2.el 361 kB/s | 27 kB 00:00 2024-03-16T20:23:20.081 INFO:teuthology.orchestra.run.smithi023.stdout:(4/174): ceph-base-19.0.0-1681.gaba3c1d2.el9.x8 2.7 MB/s | 5.3 MB 00:02 2024-03-16T20:23:20.087 INFO:teuthology.orchestra.run.smithi031.stdout:(13/174): libcephfs-devel-19.0.0-1681.gaba3c1d2 629 kB/s | 31 kB 00:00 2024-03-16T20:23:20.121 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-16T20:23:20.421 INFO:teuthology.orchestra.run.smithi023.stdout:(5/174): ceph-mgr-19.0.0-1681.gaba3c1d2.el9.x86 4.4 MB/s | 1.5 MB 00:00 2024-03-16T20:23:20.446 INFO:teuthology.orchestra.run.smithi031.stdout:(14/174): ceph-radosgw-19.0.0-1681.gaba3c1d2.el 5.4 MB/s | 9.8 MB 00:01 2024-03-16T20:23:20.513 INFO:teuthology.orchestra.run.smithi031.stdout:(15/174): libcephfs2-19.0.0-1681.gaba3c1d2.el9. 1.7 MB/s | 744 kB 00:00 2024-03-16T20:23:20.538 INFO:teuthology.orchestra.run.smithi031.stdout:(16/174): libcephsqlite-19.0.0-1681.gaba3c1d2.e 1.7 MB/s | 163 kB 00:00 2024-03-16T20:23:20.580 INFO:teuthology.orchestra.run.smithi031.stdout:(17/174): librados-devel-19.0.0-1681.gaba3c1d2. 1.9 MB/s | 127 kB 00:00 2024-03-16T20:23:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:(18/174): libradosstriper1-19.0.0-1681.gaba3c1d 3.2 MB/s | 498 kB 00:00 2024-03-16T20:23:20.781 INFO:teuthology.orchestra.run.smithi031.stdout:(19/174): python3-ceph-argparse-19.0.0-1681.gab 493 kB/s | 45 kB 00:00 2024-03-16T20:23:20.881 INFO:teuthology.orchestra.run.smithi031.stdout:(20/174): python3-ceph-common-19.0.0-1681.gaba3 1.3 MB/s | 131 kB 00:00 2024-03-16T20:23:20.981 INFO:teuthology.orchestra.run.smithi031.stdout:(21/174): python3-cephfs-19.0.0-1681.gaba3c1d2. 1.6 MB/s | 161 kB 00:00 2024-03-16T20:23:21.098 INFO:teuthology.orchestra.run.smithi031.stdout:(22/174): python3-rados-19.0.0-1681.gaba3c1d2.e 2.7 MB/s | 321 kB 00:00 2024-03-16T20:23:21.223 INFO:teuthology.orchestra.run.smithi031.stdout:(23/174): librgw2-19.0.0-1681.gaba3c1d2.el9.x86 8.0 MB/s | 5.1 MB 00:00 2024-03-16T20:23:21.249 INFO:teuthology.orchestra.run.smithi031.stdout:(24/174): python3-rbd-19.0.0-1681.gaba3c1d2.el9 1.9 MB/s | 298 kB 00:00 2024-03-16T20:23:21.282 INFO:teuthology.orchestra.run.smithi031.stdout:(25/174): python3-rgw-19.0.0-1681.gaba3c1d2.el9 1.7 MB/s | 99 kB 00:00 2024-03-16T20:23:21.298 INFO:teuthology.orchestra.run.smithi023.stdout:(6/174): ceph-mon-19.0.0-1681.gaba3c1d2.el9.x86 5.3 MB/s | 4.7 MB 00:00 2024-03-16T20:23:21.307 INFO:teuthology.orchestra.run.smithi031.stdout:(26/174): rbd-fuse-19.0.0-1681.gaba3c1d2.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-16T20:23:21.408 INFO:teuthology.orchestra.run.smithi031.stdout:(27/174): rbd-nbd-19.0.0-1681.gaba3c1d2.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-16T20:23:21.508 INFO:teuthology.orchestra.run.smithi031.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1681.g 314 kB/s | 31 kB 00:00 2024-03-16T20:23:21.539 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-03-16T20:23:21.609 INFO:teuthology.orchestra.run.smithi031.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1681.gaba3c1d 1.4 MB/s | 138 kB 00:00 2024-03-16T20:23:21.667 INFO:teuthology.orchestra.run.smithi031.stdout:(30/174): rbd-mirror-19.0.0-1681.gaba3c1d2.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-16T20:23:21.968 INFO:teuthology.orchestra.run.smithi031.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1681.gaba3c 5.0 MB/s | 1.8 MB 00:00 2024-03-16T20:23:21.989 INFO:teuthology.orchestra.run.smithi023.stdout:(7/174): ceph-mds-19.0.0-1681.gaba3c1d2.el9.x86 973 kB/s | 2.3 MB 00:02 2024-03-16T20:23:22.059 INFO:teuthology.orchestra.run.smithi031.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1681.gab 2.7 MB/s | 248 kB 00:00 2024-03-16T20:23:22.143 INFO:teuthology.orchestra.run.smithi031.stdout:(33/174): ceph-mgr-rook-19.0.0-1681.gaba3c1d2.e 592 kB/s | 49 kB 00:00 2024-03-16T20:23:22.235 INFO:teuthology.orchestra.run.smithi031.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1681.ga 167 kB/s | 15 kB 00:00 2024-03-16T20:23:22.335 INFO:teuthology.orchestra.run.smithi031.stdout:(35/174): ceph-volume-19.0.0-1681.gaba3c1d2.el9 2.6 MB/s | 264 kB 00:00 2024-03-16T20:23:22.502 INFO:teuthology.orchestra.run.smithi031.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.9 MB/s | 7.4 MB 00:00 2024-03-16T20:23:22.544 INFO:teuthology.orchestra.run.smithi031.stdout:(37/174): cephfs-top-19.0.0-1681.gaba3c1d2.el9. 440 kB/s | 18 kB 00:00 2024-03-16T20:23:22.595 INFO:teuthology.orchestra.run.smithi031.stdout:(38/174): cephadm-19.0.0-1681.gaba3c1d2.el9.noa 2.8 MB/s | 748 kB 00:00 2024-03-16T20:23:22.787 INFO:teuthology.orchestra.run.smithi031.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-16T20:23:22.845 INFO:teuthology.orchestra.run.smithi031.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-16T20:23:22.870 INFO:teuthology.orchestra.run.smithi031.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-03-16T20:23:22.904 INFO:teuthology.orchestra.run.smithi031.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-16T20:23:22.921 INFO:teuthology.orchestra.run.smithi031.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-03-16T20:23:23.013 INFO:teuthology.orchestra.run.smithi031.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-16T20:23:23.063 INFO:teuthology.orchestra.run.smithi031.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-16T20:23:23.081 INFO:teuthology.orchestra.run.smithi031.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 153 kB/s | 10 kB 00:00 2024-03-16T20:23:23.105 INFO:teuthology.orchestra.run.smithi031.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-16T20:23:23.131 INFO:teuthology.orchestra.run.smithi031.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-16T20:23:23.156 INFO:teuthology.orchestra.run.smithi031.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-16T20:23:23.198 INFO:teuthology.orchestra.run.smithi031.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-16T20:23:23.223 INFO:teuthology.orchestra.run.smithi031.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 694 kB/s | 46 kB 00:00 2024-03-16T20:23:23.257 INFO:teuthology.orchestra.run.smithi031.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-16T20:23:23.282 INFO:teuthology.orchestra.run.smithi031.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-03-16T20:23:23.308 INFO:teuthology.orchestra.run.smithi031.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T20:23:23.333 INFO:teuthology.orchestra.run.smithi031.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-16T20:23:23.467 INFO:teuthology.orchestra.run.smithi031.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-16T20:23:23.508 INFO:teuthology.orchestra.run.smithi031.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-03-16T20:23:23.542 INFO:teuthology.orchestra.run.smithi031.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-16T20:23:23.567 INFO:teuthology.orchestra.run.smithi031.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-16T20:23:23.731 INFO:teuthology.orchestra.run.smithi023.stdout:(8/174): ceph-osd-19.0.0-1681.gaba3c1d2.el9.x86 6.9 MB/s | 17 MB 00:02 2024-03-16T20:23:23.734 INFO:teuthology.orchestra.run.smithi031.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-16T20:23:23.786 INFO:teuthology.orchestra.run.smithi031.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-03-16T20:23:23.810 INFO:teuthology.orchestra.run.smithi031.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-03-16T20:23:23.931 INFO:teuthology.orchestra.run.smithi023.stdout:(9/174): ceph-common-19.0.0-1681.gaba3c1d2.el9. 3.5 MB/s | 20 MB 00:05 2024-03-16T20:23:23.957 INFO:teuthology.orchestra.run.smithi023.stdout:(10/174): ceph-selinux-19.0.0-1681.gaba3c1d2.el 112 kB/s | 25 kB 00:00 2024-03-16T20:23:24.003 INFO:teuthology.orchestra.run.smithi031.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-16T20:23:24.007 INFO:teuthology.orchestra.run.smithi023.stdout:(11/174): cephfs-mirror-19.0.0-1681.gaba3c1d2.e 4.3 MB/s | 218 kB 00:00 2024-03-16T20:23:24.020 INFO:teuthology.orchestra.run.smithi031.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 117 kB/s | 24 kB 00:00 2024-03-16T20:23:24.037 INFO:teuthology.orchestra.run.smithi031.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 538 kB/s | 18 kB 00:00 2024-03-16T20:23:24.062 INFO:teuthology.orchestra.run.smithi031.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 144 kB 00:00 2024-03-16T20:23:24.065 INFO:teuthology.orchestra.run.smithi023.stdout:(12/174): cephfs-shell-19.0.0-1681.gaba3c1d2.el 465 kB/s | 27 kB 00:00 2024-03-16T20:23:24.079 INFO:teuthology.orchestra.run.smithi031.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 759 kB/s | 32 kB 00:00 2024-03-16T20:23:24.124 INFO:teuthology.orchestra.run.smithi023.stdout:(13/174): libcephfs-devel-19.0.0-1681.gaba3c1d2 536 kB/s | 31 kB 00:00 2024-03-16T20:23:24.224 INFO:teuthology.orchestra.run.smithi023.stdout:(14/174): libcephfs2-19.0.0-1681.gaba3c1d2.el9. 7.3 MB/s | 744 kB 00:00 2024-03-16T20:23:24.246 INFO:teuthology.orchestra.run.smithi031.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-16T20:23:24.271 INFO:teuthology.orchestra.run.smithi031.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-16T20:23:24.274 INFO:teuthology.orchestra.run.smithi023.stdout:(15/174): libcephsqlite-19.0.0-1681.gaba3c1d2.e 3.2 MB/s | 163 kB 00:00 2024-03-16T20:23:24.297 INFO:teuthology.orchestra.run.smithi031.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-16T20:23:24.322 INFO:teuthology.orchestra.run.smithi031.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-16T20:23:24.339 INFO:teuthology.orchestra.run.smithi031.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 982 kB/s | 41 kB 00:00 2024-03-16T20:23:24.350 INFO:teuthology.orchestra.run.smithi023.stdout:(16/174): librados-devel-19.0.0-1681.gaba3c1d2. 1.7 MB/s | 127 kB 00:00 2024-03-16T20:23:24.381 INFO:teuthology.orchestra.run.smithi031.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-16T20:23:24.406 INFO:teuthology.orchestra.run.smithi031.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-16T20:23:24.425 INFO:teuthology.orchestra.run.smithi023.stdout:(17/174): libradosstriper1-19.0.0-1681.gaba3c1d 6.5 MB/s | 498 kB 00:00 2024-03-16T20:23:24.815 INFO:teuthology.orchestra.run.smithi031.stdout:(75/174): ceph-test-19.0.0-1681.gaba3c1d2.el9.x 7.9 MB/s | 47 MB 00:06 2024-03-16T20:23:24.841 INFO:teuthology.orchestra.run.smithi031.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 459 kB/s | 211 kB 00:00 2024-03-16T20:23:24.866 INFO:teuthology.orchestra.run.smithi031.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 103 kB/s | 47 kB 00:00 2024-03-16T20:23:24.891 INFO:teuthology.orchestra.run.smithi031.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 574 kB/s | 43 kB 00:00 2024-03-16T20:23:24.898 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-03-16T20:23:24.916 INFO:teuthology.orchestra.run.smithi031.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 775 kB/s | 39 kB 00:00 2024-03-16T20:23:24.917 INFO:teuthology.orchestra.run.smithi023.stdout:(18/174): librgw2-19.0.0-1681.gaba3c1d2.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-16T20:23:24.959 INFO:teuthology.orchestra.run.smithi023.stdout:(19/174): python3-ceph-argparse-19.0.0-1681.gab 1.1 MB/s | 45 kB 00:00 2024-03-16T20:23:25.009 INFO:teuthology.orchestra.run.smithi023.stdout:(20/174): python3-ceph-common-19.0.0-1681.gaba3 2.6 MB/s | 131 kB 00:00 2024-03-16T20:23:25.060 INFO:teuthology.orchestra.run.smithi023.stdout:(21/174): python3-cephfs-19.0.0-1681.gaba3c1d2. 3.2 MB/s | 161 kB 00:00 2024-03-16T20:23:25.126 INFO:teuthology.orchestra.run.smithi023.stdout:(22/174): python3-rados-19.0.0-1681.gaba3c1d2.e 4.7 MB/s | 321 kB 00:00 2024-03-16T20:23:25.166 INFO:teuthology.orchestra.run.smithi031.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 149 kB/s | 41 kB 00:00 2024-03-16T20:23:25.192 INFO:teuthology.orchestra.run.smithi031.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-16T20:23:25.193 INFO:teuthology.orchestra.run.smithi023.stdout:(23/174): python3-rbd-19.0.0-1681.gaba3c1d2.el9 4.4 MB/s | 298 kB 00:00 2024-03-16T20:23:25.244 INFO:teuthology.orchestra.run.smithi023.stdout:(24/174): python3-rgw-19.0.0-1681.gaba3c1d2.el9 1.9 MB/s | 99 kB 00:00 2024-03-16T20:23:25.286 INFO:teuthology.orchestra.run.smithi023.stdout:(25/174): rbd-fuse-19.0.0-1681.gaba3c1d2.el9.x8 2.0 MB/s | 85 kB 00:00 2024-03-16T20:23:25.384 INFO:teuthology.orchestra.run.smithi031.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-16T20:23:25.409 INFO:teuthology.orchestra.run.smithi031.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 233 kB/s | 56 kB 00:00 2024-03-16T20:23:25.434 INFO:teuthology.orchestra.run.smithi031.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 262 kB/s | 63 kB 00:00 2024-03-16T20:23:25.459 INFO:teuthology.orchestra.run.smithi031.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-16T20:23:25.485 INFO:teuthology.orchestra.run.smithi031.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T20:23:25.510 INFO:teuthology.orchestra.run.smithi031.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T20:23:25.560 INFO:teuthology.orchestra.run.smithi031.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-16T20:23:25.585 INFO:teuthology.orchestra.run.smithi031.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-16T20:23:25.586 INFO:teuthology.orchestra.run.smithi023.stdout:(26/174): rbd-mirror-19.0.0-1681.gaba3c1d2.el9. 10 MB/s | 3.1 MB 00:00 2024-03-16T20:23:25.610 INFO:teuthology.orchestra.run.smithi031.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-03-16T20:23:25.636 INFO:teuthology.orchestra.run.smithi031.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-16T20:23:25.636 INFO:teuthology.orchestra.run.smithi023.stdout:(27/174): rbd-nbd-19.0.0-1681.gaba3c1d2.el9.x86 3.3 MB/s | 171 kB 00:00 2024-03-16T20:23:25.670 INFO:teuthology.orchestra.run.smithi023.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1681.g 948 kB/s | 31 kB 00:00 2024-03-16T20:23:25.703 INFO:teuthology.orchestra.run.smithi031.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-03-16T20:23:25.720 INFO:teuthology.orchestra.run.smithi031.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-16T20:23:25.720 INFO:teuthology.orchestra.run.smithi023.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1681.gaba3c1d 2.7 MB/s | 138 kB 00:00 2024-03-16T20:23:25.753 INFO:teuthology.orchestra.run.smithi031.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.5 MB/s | 803 kB 00:00 2024-03-16T20:23:25.795 INFO:teuthology.orchestra.run.smithi031.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-16T20:23:25.912 INFO:teuthology.orchestra.run.smithi023.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1681.gaba3c 9.3 MB/s | 1.8 MB 00:00 2024-03-16T20:23:26.512 INFO:teuthology.orchestra.run.smithi031.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 27 kB/s | 21 kB 00:00 2024-03-16T20:23:26.537 INFO:teuthology.orchestra.run.smithi031.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 32 kB/s | 25 kB 00:00 2024-03-16T20:23:26.546 INFO:teuthology.orchestra.run.smithi023.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-16T20:23:26.687 INFO:teuthology.orchestra.run.smithi023.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1681.gab 1.7 MB/s | 248 kB 00:00 2024-03-16T20:23:26.729 INFO:teuthology.orchestra.run.smithi023.stdout:(33/174): ceph-mgr-rook-19.0.0-1681.gaba3c1d2.e 1.2 MB/s | 49 kB 00:00 2024-03-16T20:23:26.788 INFO:teuthology.orchestra.run.smithi023.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1681.ga 261 kB/s | 15 kB 00:00 2024-03-16T20:23:26.847 INFO:teuthology.orchestra.run.smithi023.stdout:(35/174): ceph-volume-19.0.0-1681.gaba3c1d2.el9 4.4 MB/s | 264 kB 00:00 2024-03-16T20:23:26.947 INFO:teuthology.orchestra.run.smithi023.stdout:(36/174): cephadm-19.0.0-1681.gaba3c1d2.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-16T20:23:26.997 INFO:teuthology.orchestra.run.smithi023.stdout:(37/174): cephfs-top-19.0.0-1681.gaba3c1d2.el9. 367 kB/s | 18 kB 00:00 2024-03-16T20:23:27.281 INFO:teuthology.orchestra.run.smithi023.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-16T20:23:27.373 INFO:teuthology.orchestra.run.smithi023.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.8 MB/s | 920 kB 00:00 2024-03-16T20:23:27.415 INFO:teuthology.orchestra.run.smithi023.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-16T20:23:27.474 INFO:teuthology.orchestra.run.smithi023.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-16T20:23:27.524 INFO:teuthology.orchestra.run.smithi023.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-16T20:23:27.629 INFO:teuthology.orchestra.run.smithi031.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 220 kB/s | 245 kB 00:01 2024-03-16T20:23:27.671 INFO:teuthology.orchestra.run.smithi031.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 219 kB/s | 248 kB 00:01 2024-03-16T20:23:27.707 INFO:teuthology.orchestra.run.smithi023.stdout:(43/174): ceph-radosgw-19.0.0-1681.gaba3c1d2.el 1.7 MB/s | 9.8 MB 00:05 2024-03-16T20:23:27.766 INFO:teuthology.orchestra.run.smithi023.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-16T20:23:27.793 INFO:teuthology.orchestra.run.smithi023.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 212 kB/s | 17 kB 00:00 2024-03-16T20:23:27.817 INFO:teuthology.orchestra.run.smithi023.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 202 kB/s | 10 kB 00:00 2024-03-16T20:23:27.842 INFO:teuthology.orchestra.run.smithi023.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T20:23:27.875 INFO:teuthology.orchestra.run.smithi023.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-03-16T20:23:27.909 INFO:teuthology.orchestra.run.smithi023.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-16T20:23:27.943 INFO:teuthology.orchestra.run.smithi023.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-16T20:23:27.968 INFO:teuthology.orchestra.run.smithi023.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 793 kB/s | 46 kB 00:00 2024-03-16T20:23:28.002 INFO:teuthology.orchestra.run.smithi023.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-16T20:23:28.018 INFO:teuthology.orchestra.run.smithi023.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-16T20:23:28.061 INFO:teuthology.orchestra.run.smithi023.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-16T20:23:28.086 INFO:teuthology.orchestra.run.smithi023.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 510 kB/s | 34 kB 00:00 2024-03-16T20:23:28.286 INFO:teuthology.orchestra.run.smithi023.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-16T20:23:28.328 INFO:teuthology.orchestra.run.smithi023.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-03-16T20:23:28.361 INFO:teuthology.orchestra.run.smithi023.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-16T20:23:28.387 INFO:teuthology.orchestra.run.smithi023.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-16T20:23:28.438 INFO:teuthology.orchestra.run.smithi031.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 45 kB/s | 35 kB 00:00 2024-03-16T20:23:28.587 INFO:teuthology.orchestra.run.smithi023.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-16T20:23:28.621 INFO:teuthology.orchestra.run.smithi023.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-16T20:23:28.637 INFO:teuthology.orchestra.run.smithi023.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-16T20:23:28.730 INFO:teuthology.orchestra.run.smithi031.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 197 kB/s | 216 kB 00:01 2024-03-16T20:23:29.222 INFO:teuthology.orchestra.run.smithi023.stdout:(63/174): ceph-test-19.0.0-1681.gaba3c1d2.el9.x 9.0 MB/s | 47 MB 00:05 2024-03-16T20:23:29.230 INFO:teuthology.orchestra.run.smithi031.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 56 kB/s | 44 kB 00:00 2024-03-16T20:23:29.297 INFO:teuthology.orchestra.run.smithi023.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 8.6 MB/s | 5.8 MB 00:00 2024-03-16T20:23:29.313 INFO:teuthology.orchestra.run.smithi023.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 36 kB/s | 24 kB 00:00 2024-03-16T20:23:29.330 INFO:teuthology.orchestra.run.smithi023.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 171 kB/s | 18 kB 00:00 2024-03-16T20:23:29.364 INFO:teuthology.orchestra.run.smithi023.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-16T20:23:29.389 INFO:teuthology.orchestra.run.smithi023.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-16T20:23:29.539 INFO:teuthology.orchestra.run.smithi023.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-16T20:23:29.547 INFO:teuthology.orchestra.run.smithi031.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-03-16T20:23:29.614 INFO:teuthology.orchestra.run.smithi031.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 106 kB/s | 94 kB 00:00 2024-03-16T20:23:29.641 INFO:teuthology.orchestra.run.smithi023.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 788 kB/s | 217 kB 00:00 2024-03-16T20:23:29.715 INFO:teuthology.orchestra.run.smithi023.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 540 kB/s | 174 kB 00:00 2024-03-16T20:23:29.740 INFO:teuthology.orchestra.run.smithi031.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-03-16T20:23:29.740 INFO:teuthology.orchestra.run.smithi023.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 344 kB/s | 69 kB 00:00 2024-03-16T20:23:29.765 INFO:teuthology.orchestra.run.smithi023.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-16T20:23:29.791 INFO:teuthology.orchestra.run.smithi023.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T20:23:29.816 INFO:teuthology.orchestra.run.smithi023.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-16T20:23:29.823 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-16T20:23:29.841 INFO:teuthology.orchestra.run.smithi023.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 955 kB/s | 47 kB 00:00 2024-03-16T20:23:29.866 INFO:teuthology.orchestra.run.smithi023.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-16T20:23:29.892 INFO:teuthology.orchestra.run.smithi023.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-16T20:23:30.223 INFO:teuthology.orchestra.run.smithi031.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 901 kB/s | 548 kB 00:00 2024-03-16T20:23:30.290 INFO:teuthology.orchestra.run.smithi031.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 561 kB/s | 309 kB 00:00 2024-03-16T20:23:30.384 INFO:teuthology.orchestra.run.smithi023.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-16T20:23:30.409 INFO:teuthology.orchestra.run.smithi023.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 75 kB/s | 39 kB 00:00 2024-03-16T20:23:30.434 INFO:teuthology.orchestra.run.smithi023.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 76 kB/s | 41 kB 00:00 2024-03-16T20:23:30.457 INFO:teuthology.orchestra.run.smithi031.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-16T20:23:30.459 INFO:teuthology.orchestra.run.smithi023.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-16T20:23:30.484 INFO:teuthology.orchestra.run.smithi023.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-16T20:23:30.510 INFO:teuthology.orchestra.run.smithi023.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2024-03-16T20:23:30.535 INFO:teuthology.orchestra.run.smithi023.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-16T20:23:30.540 INFO:teuthology.orchestra.run.smithi031.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-16T20:23:30.560 INFO:teuthology.orchestra.run.smithi023.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-16T20:23:30.585 INFO:teuthology.orchestra.run.smithi023.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T20:23:30.619 INFO:teuthology.orchestra.run.smithi023.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-16T20:23:30.636 INFO:teuthology.orchestra.run.smithi023.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-16T20:23:30.649 INFO:teuthology.orchestra.run.smithi031.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-03-16T20:23:30.661 INFO:teuthology.orchestra.run.smithi023.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-16T20:23:30.686 INFO:teuthology.orchestra.run.smithi023.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-16T20:23:30.732 INFO:teuthology.orchestra.run.smithi031.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-16T20:23:30.800 INFO:teuthology.orchestra.run.smithi031.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-03-16T20:23:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-16T20:23:30.900 INFO:teuthology.orchestra.run.smithi031.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-16T20:23:30.971 INFO:teuthology.orchestra.run.smithi031.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 615 kB/s | 43 kB 00:00 2024-03-16T20:23:30.996 INFO:teuthology.orchestra.run.smithi031.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-03-16T20:23:31.038 INFO:teuthology.orchestra.run.smithi031.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-03-16T20:23:31.064 INFO:teuthology.orchestra.run.smithi031.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-16T20:23:31.131 INFO:teuthology.orchestra.run.smithi031.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-16T20:23:31.156 INFO:teuthology.orchestra.run.smithi031.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-16T20:23:31.214 INFO:teuthology.orchestra.run.smithi031.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-16T20:23:31.240 INFO:teuthology.orchestra.run.smithi031.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 423 kB/s | 35 kB 00:00 2024-03-16T20:23:31.307 INFO:teuthology.orchestra.run.smithi031.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-16T20:23:31.320 INFO:teuthology.orchestra.run.smithi023.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 308 kB/s | 195 kB 00:00 2024-03-16T20:23:31.474 INFO:teuthology.orchestra.run.smithi031.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-03-16T20:23:31.495 INFO:teuthology.orchestra.run.smithi023.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 714 kB/s | 125 kB 00:00 2024-03-16T20:23:31.499 INFO:teuthology.orchestra.run.smithi031.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 753 kB/s | 144 kB 00:00 2024-03-16T20:23:31.549 INFO:teuthology.orchestra.run.smithi031.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-16T20:23:31.583 INFO:teuthology.orchestra.run.smithi031.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-16T20:23:31.616 INFO:teuthology.orchestra.run.smithi031.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 345 kB/s | 23 kB 00:00 2024-03-16T20:23:31.650 INFO:teuthology.orchestra.run.smithi031.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-03-16T20:23:31.662 INFO:teuthology.orchestra.run.smithi023.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 129 kB/s | 21 kB 00:00 2024-03-16T20:23:31.700 INFO:teuthology.orchestra.run.smithi031.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-16T20:23:31.720 INFO:teuthology.orchestra.run.smithi023.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-03-16T20:23:31.776 INFO:teuthology.orchestra.run.smithi031.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-03-16T20:23:31.801 INFO:teuthology.orchestra.run.smithi031.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-03-16T20:23:31.859 INFO:teuthology.orchestra.run.smithi031.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 946 kB/s | 79 kB 00:00 2024-03-16T20:23:31.885 INFO:teuthology.orchestra.run.smithi031.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 691 kB/s | 58 kB 00:00 2024-03-16T20:23:31.935 INFO:teuthology.orchestra.run.smithi031.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-16T20:23:31.960 INFO:teuthology.orchestra.run.smithi031.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-03-16T20:23:31.962 INFO:teuthology.orchestra.run.smithi023.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-03-16T20:23:32.011 INFO:teuthology.orchestra.run.smithi031.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-16T20:23:32.030 INFO:teuthology.orchestra.run.smithi023.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-16T20:23:32.036 INFO:teuthology.orchestra.run.smithi031.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-16T20:23:32.086 INFO:teuthology.orchestra.run.smithi031.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-16T20:23:32.096 INFO:teuthology.orchestra.run.smithi023.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-03-16T20:23:32.111 INFO:teuthology.orchestra.run.smithi031.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T20:23:32.162 INFO:teuthology.orchestra.run.smithi031.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-16T20:23:32.163 INFO:teuthology.orchestra.run.smithi023.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-16T20:23:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-03-16T20:23:32.222 INFO:teuthology.orchestra.run.smithi023.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 601 kB/s | 35 kB 00:00 2024-03-16T20:23:32.237 INFO:teuthology.orchestra.run.smithi031.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 680 kB/s | 51 kB 00:00 2024-03-16T20:23:32.276 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:32.277 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-16T20:23:32.279 INFO:teuthology.orchestra.run.smithi031.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-16T20:23:32.281 INFO:teuthology.orchestra.run.smithi023.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 762 kB/s | 44 kB 00:00 2024-03-16T20:23:32.289 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-16T20:23:32.290 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-16T20:23:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-16T20:23:32.291 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:32.293 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-16T20:23:32.296 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-16T20:23:32.297 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-16T20:23:32.306 INFO:teuthology.orchestra.run.smithi023.stdout:(102/174): jq-1.6-16.el9.x86_64.rpm 114 kB/s | 187 kB 00:01 2024-03-16T20:23:32.313 INFO:teuthology.orchestra.run.smithi031.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T20:23:32.348 INFO:teuthology.orchestra.run.smithi023.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-16T20:23:32.363 INFO:teuthology.orchestra.run.smithi031.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-16T20:23:32.376 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-16T20:23:32.388 INFO:teuthology.orchestra.run.smithi031.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-16T20:23:32.395 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 6.6 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.3 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 887 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 145 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 1.5 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 138 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 1.8 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 7.4 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 49 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 9.8 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 47 M 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 264 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 748 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 218 k 2024-03-16T20:23:32.396 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 27 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 18 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 31 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 744 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 127 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-16T20:23:32.397 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 161 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 321 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 298 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 99 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 85 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 171 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-16T20:23:32.398 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.4 M 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 3.1 M 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-16T20:23:32.399 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 20 M 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 31 k 2024-03-16T20:23:32.400 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 2.3 M 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 248 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 4.7 M 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 17 M 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1681.gaba3c1d2.el9 ceph-noarch 15 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 25 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-16T20:23:32.401 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 163 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 498 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-16T20:23:32.402 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 5.1 M 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-16T20:23:32.403 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 45 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-1681.gaba3c1d2.el9 ceph 131 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-16T20:23:32.404 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-16T20:23:32.405 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-16T20:23:32.406 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-16T20:23:32.407 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-16T20:23:32.408 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout:Install 150 Packages 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 24 Packages 2024-03-16T20:23:32.409 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:23:32.410 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 224 M 2024-03-16T20:23:32.410 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-16T20:23:32.415 INFO:teuthology.orchestra.run.smithi023.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-16T20:23:32.430 INFO:teuthology.orchestra.run.smithi031.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-16T20:23:32.514 INFO:teuthology.orchestra.run.smithi031.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-16T20:23:32.565 INFO:teuthology.orchestra.run.smithi023.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-03-16T20:23:32.724 INFO:teuthology.orchestra.run.smithi023.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-16T20:23:32.782 INFO:teuthology.orchestra.run.smithi023.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 336 kB/s | 19 kB 00:00 2024-03-16T20:23:32.792 INFO:teuthology.orchestra.run.smithi189.stdout:(1/174): ceph-19.0.0-1681.gaba3c1d2.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-16T20:23:32.808 INFO:teuthology.orchestra.run.smithi023.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-16T20:23:32.833 INFO:teuthology.orchestra.run.smithi023.stdout:(109/174): libgfortran-11.4.1-3.el9.x86_64.rpm 370 kB/s | 803 kB 00:02 2024-03-16T20:23:32.866 INFO:teuthology.orchestra.run.smithi023.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-16T20:23:32.889 INFO:teuthology.orchestra.run.smithi031.stdout:(150/174): openssl-devel-3.0.7-27.el9.x86_64.rp 596 kB/s | 4.1 MB 00:07 2024-03-16T20:23:32.925 INFO:teuthology.orchestra.run.smithi023.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-16T20:23:32.939 INFO:teuthology.orchestra.run.smithi031.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-16T20:23:33.000 INFO:teuthology.orchestra.run.smithi023.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 290 kB/s | 49 kB 00:00 2024-03-16T20:23:33.017 INFO:teuthology.orchestra.run.smithi189.stdout:(2/174): ceph-fuse-19.0.0-1681.gaba3c1d2.el9.x8 3.9 MB/s | 887 kB 00:00 2024-03-16T20:23:33.059 INFO:teuthology.orchestra.run.smithi023.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-16T20:23:33.084 INFO:teuthology.orchestra.run.smithi189.stdout:(3/174): ceph-immutable-object-cache-19.0.0-168 2.1 MB/s | 145 kB 00:00 2024-03-16T20:23:33.084 INFO:teuthology.orchestra.run.smithi023.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 951 kB/s | 151 kB 00:00 2024-03-16T20:23:33.118 INFO:teuthology.orchestra.run.smithi023.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-16T20:23:33.168 INFO:teuthology.orchestra.run.smithi023.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-03-16T20:23:33.173 INFO:teuthology.orchestra.run.smithi031.stdout:(152/174): librbd1-19.0.0-1681.gaba3c1d2.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-03-16T20:23:33.193 INFO:teuthology.orchestra.run.smithi023.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-16T20:23:33.218 INFO:teuthology.orchestra.run.smithi023.stdout:(118/174): python3-asyncssh-2.13.2-4.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-16T20:23:33.223 INFO:teuthology.orchestra.run.smithi031.stdout:(153/174): librados2-19.0.0-1681.gaba3c1d2.el9. 4.3 MB/s | 3.4 MB 00:00 2024-03-16T20:23:33.234 INFO:teuthology.orchestra.run.smithi189.stdout:(4/174): ceph-base-19.0.0-1681.gaba3c1d2.el9.x8 8.9 MB/s | 5.3 MB 00:00 2024-03-16T20:23:33.244 INFO:teuthology.orchestra.run.smithi023.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-16T20:23:33.269 INFO:teuthology.orchestra.run.smithi023.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-16T20:23:33.282 INFO:teuthology.orchestra.run.smithi031.stdout:(154/174): NetworkManager-1.46.0-1.el9.x86_64.r 6.7 MB/s | 2.3 MB 00:00 2024-03-16T20:23:33.294 INFO:teuthology.orchestra.run.smithi023.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-16T20:23:33.319 INFO:teuthology.orchestra.run.smithi023.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-03-16T20:23:33.323 INFO:teuthology.orchestra.run.smithi031.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 423 kB/s | 42 kB 00:00 2024-03-16T20:23:33.351 INFO:teuthology.orchestra.run.smithi189.stdout:(5/174): ceph-mds-19.0.0-1681.gaba3c1d2.el9.x86 8.6 MB/s | 2.3 MB 00:00 2024-03-16T20:23:33.353 INFO:teuthology.orchestra.run.smithi023.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-16T20:23:33.365 INFO:teuthology.orchestra.run.smithi031.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-16T20:23:33.378 INFO:teuthology.orchestra.run.smithi023.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-16T20:23:33.385 INFO:teuthology.orchestra.run.smithi189.stdout:(6/174): ceph-mgr-19.0.0-1681.gaba3c1d2.el9.x86 9.9 MB/s | 1.5 MB 00:00 2024-03-16T20:23:33.407 INFO:teuthology.orchestra.run.smithi031.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.8 MB/s | 1.8 MB 00:00 2024-03-16T20:23:33.420 INFO:teuthology.orchestra.run.smithi023.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-16T20:23:33.424 INFO:teuthology.orchestra.run.smithi031.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 708 kB/s | 71 kB 00:00 2024-03-16T20:23:33.446 INFO:teuthology.orchestra.run.smithi023.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-16T20:23:33.450 INFO:teuthology.orchestra.run.smithi031.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 434 kB/s | 36 kB 00:00 2024-03-16T20:23:33.471 INFO:teuthology.orchestra.run.smithi023.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-16T20:23:33.488 INFO:teuthology.orchestra.run.smithi023.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 262 kB/s | 18 kB 00:00 2024-03-16T20:23:33.491 INFO:teuthology.orchestra.run.smithi031.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-16T20:23:33.505 INFO:teuthology.orchestra.run.smithi023.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 392 kB/s | 23 kB 00:00 2024-03-16T20:23:33.517 INFO:teuthology.orchestra.run.smithi031.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 345 kB/s | 23 kB 00:00 2024-03-16T20:23:33.522 INFO:teuthology.orchestra.run.smithi023.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 383 kB/s | 19 kB 00:00 2024-03-16T20:23:33.539 INFO:teuthology.orchestra.run.smithi023.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 389 kB/s | 20 kB 00:00 2024-03-16T20:23:33.542 INFO:teuthology.orchestra.run.smithi031.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-16T20:23:33.573 INFO:teuthology.orchestra.run.smithi023.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 924 kB/s | 46 kB 00:00 2024-03-16T20:23:33.576 INFO:teuthology.orchestra.run.smithi031.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-16T20:23:33.589 INFO:teuthology.orchestra.run.smithi023.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-16T20:23:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-16T20:23:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-03-16T20:23:33.643 INFO:teuthology.orchestra.run.smithi031.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 959 kB/s | 64 kB 00:00 2024-03-16T20:23:33.656 INFO:teuthology.orchestra.run.smithi023.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-03-16T20:23:33.668 INFO:teuthology.orchestra.run.smithi031.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-16T20:23:33.690 INFO:teuthology.orchestra.run.smithi023.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-16T20:23:33.693 INFO:teuthology.orchestra.run.smithi031.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-16T20:23:33.715 INFO:teuthology.orchestra.run.smithi023.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-16T20:23:33.740 INFO:teuthology.orchestra.run.smithi023.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-16T20:23:33.766 INFO:teuthology.orchestra.run.smithi023.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-16T20:23:33.785 INFO:teuthology.orchestra.run.smithi031.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-16T20:23:33.791 INFO:teuthology.orchestra.run.smithi023.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-16T20:23:33.816 INFO:teuthology.orchestra.run.smithi023.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-16T20:23:33.841 INFO:teuthology.orchestra.run.smithi023.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-16T20:23:33.851 INFO:teuthology.orchestra.run.smithi189.stdout:(7/174): ceph-mon-19.0.0-1681.gaba3c1d2.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-03-16T20:23:33.867 INFO:teuthology.orchestra.run.smithi023.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-03-16T20:23:33.877 INFO:teuthology.orchestra.run.smithi031.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-16T20:23:33.892 INFO:teuthology.orchestra.run.smithi023.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-16T20:23:33.917 INFO:teuthology.orchestra.run.smithi023.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-16T20:23:33.927 INFO:teuthology.orchestra.run.smithi031.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.8 MB/s | 2.3 MB 00:00 2024-03-16T20:23:33.943 INFO:teuthology.orchestra.run.smithi023.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-16T20:23:33.961 INFO:teuthology.orchestra.run.smithi031.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.6 MB/s | 465 kB 00:00 2024-03-16T20:23:33.985 INFO:teuthology.orchestra.run.smithi023.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-16T20:23:33.986 INFO:teuthology.orchestra.run.smithi031.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 286 kB/s | 31 kB 00:00 2024-03-16T20:23:34.010 INFO:teuthology.orchestra.run.smithi023.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-16T20:23:34.035 INFO:teuthology.orchestra.run.smithi023.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-16T20:23:34.061 INFO:teuthology.orchestra.run.smithi031.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-16T20:23:34.169 INFO:teuthology.orchestra.run.smithi023.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-16T20:23:34.352 INFO:teuthology.orchestra.run.smithi023.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.7 MB/s | 321 kB 00:00 2024-03-16T20:23:34.518 INFO:teuthology.orchestra.run.smithi189.stdout:(8/174): ceph-osd-19.0.0-1681.gaba3c1d2.el9.x86 15 MB/s | 17 MB 00:01 2024-03-16T20:23:34.585 INFO:teuthology.orchestra.run.smithi189.stdout:(9/174): ceph-selinux-19.0.0-1681.gaba3c1d2.el9 378 kB/s | 25 kB 00:00 2024-03-16T20:23:34.712 INFO:teuthology.orchestra.run.smithi031.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 99 kB/s | 74 kB 00:00 2024-03-16T20:23:34.722 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-03-16T20:23:34.722 INFO:teuthology.orchestra.run.smithi031.stdout:Total 12 MB/s | 224 MB 00:18 2024-03-16T20:23:34.886 INFO:teuthology.orchestra.run.smithi023.stdout:(152/174): librados2-19.0.0-1681.gaba3c1d2.el9. 3.9 MB/s | 3.4 MB 00:00 2024-03-16T20:23:34.944 INFO:teuthology.orchestra.run.smithi189.stdout:(10/174): ceph-radosgw-19.0.0-1681.gaba3c1d2.el 8.9 MB/s | 9.8 MB 00:01 2024-03-16T20:23:35.019 INFO:teuthology.orchestra.run.smithi189.stdout:(11/174): cephfs-mirror-19.0.0-1681.gaba3c1d2.e 2.8 MB/s | 218 kB 00:00 2024-03-16T20:23:35.094 INFO:teuthology.orchestra.run.smithi189.stdout:(12/174): cephfs-shell-19.0.0-1681.gaba3c1d2.el 361 kB/s | 27 kB 00:00 2024-03-16T20:23:35.170 INFO:teuthology.orchestra.run.smithi189.stdout:(13/174): libcephfs-devel-19.0.0-1681.gaba3c1d2 416 kB/s | 31 kB 00:00 2024-03-16T20:23:35.216 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-03-16T20:23:35.328 INFO:teuthology.orchestra.run.smithi189.stdout:(14/174): libcephfs2-19.0.0-1681.gaba3c1d2.el9. 4.6 MB/s | 744 kB 00:00 2024-03-16T20:23:35.404 INFO:teuthology.orchestra.run.smithi189.stdout:(15/174): libcephsqlite-19.0.0-1681.gaba3c1d2.e 2.1 MB/s | 163 kB 00:00 2024-03-16T20:23:35.487 INFO:teuthology.orchestra.run.smithi189.stdout:(16/174): librados-devel-19.0.0-1681.gaba3c1d2. 1.5 MB/s | 127 kB 00:00 2024-03-16T20:23:35.515 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-03-16T20:23:35.515 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-03-16T20:23:35.638 INFO:teuthology.orchestra.run.smithi189.stdout:(17/174): libradosstriper1-19.0.0-1681.gaba3c1d 3.2 MB/s | 498 kB 00:00 2024-03-16T20:23:35.686 INFO:teuthology.orchestra.run.smithi023.stdout:(153/174): librbd1-19.0.0-1681.gaba3c1d2.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-03-16T20:23:35.803 INFO:teuthology.orchestra.run.smithi023.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 364 kB/s | 42 kB 00:00 2024-03-16T20:23:35.920 INFO:teuthology.orchestra.run.smithi023.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-16T20:23:35.970 INFO:teuthology.orchestra.run.smithi023.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-16T20:23:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 531 kB/s | 36 kB 00:00 2024-03-16T20:23:36.095 INFO:teuthology.orchestra.run.smithi023.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.3 MB/s | 192 kB 00:00 2024-03-16T20:23:36.146 INFO:teuthology.orchestra.run.smithi023.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-16T20:23:36.196 INFO:teuthology.orchestra.run.smithi023.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 461 kB/s | 23 kB 00:00 2024-03-16T20:23:36.255 INFO:teuthology.orchestra.run.smithi023.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-16T20:23:36.372 INFO:teuthology.orchestra.run.smithi023.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-16T20:23:36.463 INFO:teuthology.orchestra.run.smithi189.stdout:(18/174): librgw2-19.0.0-1681.gaba3c1d2.el9.x86 6.2 MB/s | 5.1 MB 00:00 2024-03-16T20:23:36.489 INFO:teuthology.orchestra.run.smithi023.stdout:(163/174): NetworkManager-1.46.0-1.el9.x86_64.r 1.1 MB/s | 2.3 MB 00:02 2024-03-16T20:23:36.514 INFO:teuthology.orchestra.run.smithi023.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 956 kB/s | 136 kB 00:00 2024-03-16T20:23:36.529 INFO:teuthology.orchestra.run.smithi189.stdout:(19/174): python3-ceph-argparse-19.0.0-1681.gab 677 kB/s | 45 kB 00:00 2024-03-16T20:23:36.564 INFO:teuthology.orchestra.run.smithi023.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-16T20:23:36.590 INFO:teuthology.orchestra.run.smithi023.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-03-16T20:23:36.605 INFO:teuthology.orchestra.run.smithi189.stdout:(20/174): python3-ceph-common-19.0.0-1681.gaba3 1.7 MB/s | 131 kB 00:00 2024-03-16T20:23:36.632 INFO:teuthology.orchestra.run.smithi023.stdout:(167/174): NetworkManager-libnm-1.46.0-1.el9.x8 1.0 MB/s | 1.8 MB 00:01 2024-03-16T20:23:36.680 INFO:teuthology.orchestra.run.smithi189.stdout:(21/174): python3-cephfs-19.0.0-1681.gaba3c1d2. 2.1 MB/s | 161 kB 00:00 2024-03-16T20:23:36.749 INFO:teuthology.orchestra.run.smithi023.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-03-16T20:23:36.772 INFO:teuthology.orchestra.run.smithi189.stdout:(22/174): python3-rados-19.0.0-1681.gaba3c1d2.e 3.4 MB/s | 321 kB 00:00 2024-03-16T20:23:36.856 INFO:teuthology.orchestra.run.smithi189.stdout:(23/174): python3-rbd-19.0.0-1681.gaba3c1d2.el9 3.5 MB/s | 298 kB 00:00 2024-03-16T20:23:36.857 INFO:teuthology.orchestra.run.smithi023.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-16T20:23:36.949 INFO:teuthology.orchestra.run.smithi023.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 340 kB/s | 31 kB 00:00 2024-03-16T20:23:37.024 INFO:teuthology.orchestra.run.smithi023.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-16T20:23:37.100 INFO:teuthology.orchestra.run.smithi023.stdout:(172/174): openssl-3.0.7-27.el9.x86_64.rpm 2.3 MB/s | 1.2 MB 00:00 2024-03-16T20:23:37.158 INFO:teuthology.orchestra.run.smithi023.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-16T20:23:37.323 INFO:teuthology.orchestra.run.smithi189.stdout:(24/174): ceph-test-19.0.0-1681.gaba3c1d2.el9.x 17 MB/s | 47 MB 00:02 2024-03-16T20:23:37.337 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-03-16T20:23:37.337 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-03-16T20:23:37.340 INFO:teuthology.orchestra.run.smithi189.stdout:(25/174): python3-rgw-19.0.0-1681.gaba3c1d2.el9 205 kB/s | 99 kB 00:00 2024-03-16T20:23:37.373 INFO:teuthology.orchestra.run.smithi189.stdout:(26/174): rbd-fuse-19.0.0-1681.gaba3c1d2.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-16T20:23:37.432 INFO:teuthology.orchestra.run.smithi189.stdout:(27/174): rbd-nbd-19.0.0-1681.gaba3c1d2.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-16T20:23:37.491 INFO:teuthology.orchestra.run.smithi189.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1681.g 539 kB/s | 31 kB 00:00 2024-03-16T20:23:37.500 INFO:teuthology.orchestra.run.smithi023.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.4 MB/s | 2.2 MB 00:00 2024-03-16T20:23:37.505 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-16T20:23:37.506 INFO:teuthology.orchestra.run.smithi023.stdout:Total 11 MB/s | 224 MB 00:19 2024-03-16T20:23:37.558 INFO:teuthology.orchestra.run.smithi189.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1681.gaba3c1d 2.0 MB/s | 138 kB 00:00 2024-03-16T20:23:37.625 INFO:teuthology.orchestra.run.smithi189.stdout:(30/174): rbd-mirror-19.0.0-1681.gaba3c1d2.el9. 11 MB/s | 3.1 MB 00:00 2024-03-16T20:23:37.684 INFO:teuthology.orchestra.run.smithi189.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1681.gaba3c 14 MB/s | 1.8 MB 00:00 2024-03-16T20:23:37.750 INFO:teuthology.orchestra.run.smithi189.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1681.gab 3.7 MB/s | 248 kB 00:00 2024-03-16T20:23:37.792 INFO:teuthology.orchestra.run.smithi189.stdout:(33/174): ceph-mgr-rook-19.0.0-1681.gaba3c1d2.e 1.2 MB/s | 49 kB 00:00 2024-03-16T20:23:37.834 INFO:teuthology.orchestra.run.smithi189.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1681.ga 367 kB/s | 15 kB 00:00 2024-03-16T20:23:37.901 INFO:teuthology.orchestra.run.smithi189.stdout:(35/174): ceph-volume-19.0.0-1681.gaba3c1d2.el9 3.9 MB/s | 264 kB 00:00 2024-03-16T20:23:37.993 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-16T20:23:37.994 INFO:teuthology.orchestra.run.smithi189.stdout:(36/174): cephadm-19.0.0-1681.gaba3c1d2.el9.noa 8.0 MB/s | 748 kB 00:00 2024-03-16T20:23:38.043 INFO:teuthology.orchestra.run.smithi189.stdout:(37/174): cephfs-top-19.0.0-1681.gaba3c1d2.el9. 366 kB/s | 18 kB 00:00 2024-03-16T20:23:38.219 INFO:teuthology.orchestra.run.smithi189.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-16T20:23:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-16T20:23:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-16T20:23:38.344 INFO:teuthology.orchestra.run.smithi189.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-16T20:23:38.411 INFO:teuthology.orchestra.run.smithi189.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-16T20:23:38.428 INFO:teuthology.orchestra.run.smithi189.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 106 kB 00:00 2024-03-16T20:23:38.470 INFO:teuthology.orchestra.run.smithi189.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-16T20:23:38.487 INFO:teuthology.orchestra.run.smithi189.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 540 kB/s | 32 kB 00:00 2024-03-16T20:23:38.579 INFO:teuthology.orchestra.run.smithi189.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-16T20:23:38.654 INFO:teuthology.orchestra.run.smithi189.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-16T20:23:38.671 INFO:teuthology.orchestra.run.smithi189.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 10 kB 00:00 2024-03-16T20:23:38.704 INFO:teuthology.orchestra.run.smithi189.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-16T20:23:38.722 INFO:teuthology.orchestra.run.smithi189.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-16T20:23:38.805 INFO:teuthology.orchestra.run.smithi189.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-16T20:23:38.839 INFO:teuthology.orchestra.run.smithi189.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 578 kB 00:00 2024-03-16T20:23:38.856 INFO:teuthology.orchestra.run.smithi189.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 942 kB/s | 46 kB 00:00 2024-03-16T20:23:38.889 INFO:teuthology.orchestra.run.smithi189.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-16T20:23:38.915 INFO:teuthology.orchestra.run.smithi189.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-16T20:23:38.940 INFO:teuthology.orchestra.run.smithi189.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-16T20:23:38.974 INFO:teuthology.orchestra.run.smithi189.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 585 kB/s | 34 kB 00:00 2024-03-16T20:23:39.191 INFO:teuthology.orchestra.run.smithi189.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-03-16T20:23:39.233 INFO:teuthology.orchestra.run.smithi189.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 5.9 MB/s | 1.5 MB 00:00 2024-03-16T20:23:39.274 INFO:teuthology.orchestra.run.smithi189.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.7 MB/s | 811 kB 00:00 2024-03-16T20:23:39.300 INFO:teuthology.orchestra.run.smithi189.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-16T20:23:39.492 INFO:teuthology.orchestra.run.smithi189.stdout:(60/174): ceph-common-19.0.0-1681.gaba3c1d2.el9 3.0 MB/s | 20 MB 00:06 2024-03-16T20:23:39.584 INFO:teuthology.orchestra.run.smithi189.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-03-16T20:23:39.617 INFO:teuthology.orchestra.run.smithi189.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-03-16T20:23:39.650 INFO:teuthology.orchestra.run.smithi189.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 501 kB/s | 78 kB 00:00 2024-03-16T20:23:39.809 INFO:teuthology.orchestra.run.smithi189.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-16T20:23:39.834 INFO:teuthology.orchestra.run.smithi189.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 135 kB/s | 24 kB 00:00 2024-03-16T20:23:39.860 INFO:teuthology.orchestra.run.smithi189.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 88 kB/s | 18 kB 00:00 2024-03-16T20:23:39.885 INFO:teuthology.orchestra.run.smithi189.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-16T20:23:39.919 INFO:teuthology.orchestra.run.smithi189.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 536 kB/s | 32 kB 00:00 2024-03-16T20:23:40.103 INFO:teuthology.orchestra.run.smithi189.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-16T20:23:40.128 INFO:teuthology.orchestra.run.smithi189.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 898 kB/s | 217 kB 00:00 2024-03-16T20:23:40.161 INFO:teuthology.orchestra.run.smithi189.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 730 kB/s | 174 kB 00:00 2024-03-16T20:23:40.187 INFO:teuthology.orchestra.run.smithi189.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 828 kB/s | 69 kB 00:00 2024-03-16T20:23:40.212 INFO:teuthology.orchestra.run.smithi189.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-16T20:23:40.237 INFO:teuthology.orchestra.run.smithi189.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-16T20:23:40.262 INFO:teuthology.orchestra.run.smithi189.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-16T20:23:40.288 INFO:teuthology.orchestra.run.smithi189.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-03-16T20:23:40.313 INFO:teuthology.orchestra.run.smithi189.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-16T20:23:40.338 INFO:teuthology.orchestra.run.smithi189.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-16T20:23:40.525 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-16T20:23:40.526 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-16T20:23:40.589 INFO:teuthology.orchestra.run.smithi189.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 140 kB/s | 39 kB 00:00 2024-03-16T20:23:40.639 INFO:teuthology.orchestra.run.smithi189.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-16T20:23:40.889 INFO:teuthology.orchestra.run.smithi189.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-16T20:23:40.915 INFO:teuthology.orchestra.run.smithi189.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 928 kB/s | 300 kB 00:00 2024-03-16T20:23:40.931 INFO:teuthology.orchestra.run.smithi189.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 194 kB/s | 56 kB 00:00 2024-03-16T20:23:40.948 INFO:teuthology.orchestra.run.smithi189.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-16T20:23:40.965 INFO:teuthology.orchestra.run.smithi189.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-16T20:23:40.990 INFO:teuthology.orchestra.run.smithi189.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-16T20:23:41.024 INFO:teuthology.orchestra.run.smithi189.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-16T20:23:41.057 INFO:teuthology.orchestra.run.smithi189.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-16T20:23:41.083 INFO:teuthology.orchestra.run.smithi189.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-16T20:23:41.108 INFO:teuthology.orchestra.run.smithi189.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-16T20:23:41.133 INFO:teuthology.orchestra.run.smithi189.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-16T20:23:41.200 INFO:teuthology.orchestra.run.smithi189.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-16T20:23:41.225 INFO:teuthology.orchestra.run.smithi189.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-16T20:23:41.259 INFO:teuthology.orchestra.run.smithi189.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.2 MB/s | 803 kB 00:00 2024-03-16T20:23:41.301 INFO:teuthology.orchestra.run.smithi189.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-16T20:23:41.351 INFO:teuthology.orchestra.run.smithi189.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 171 kB/s | 21 kB 00:00 2024-03-16T20:23:41.510 INFO:teuthology.orchestra.run.smithi189.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 102 kB/s | 25 kB 00:00 2024-03-16T20:23:42.343 INFO:teuthology.orchestra.run.smithi189.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 247 kB/s | 245 kB 00:00 2024-03-16T20:23:42.752 INFO:teuthology.orchestra.run.smithi189.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 200 kB/s | 248 kB 00:01 2024-03-16T20:23:42.952 INFO:teuthology.orchestra.run.smithi189.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 174 kB/s | 35 kB 00:00 2024-03-16T20:23:43.170 INFO:teuthology.orchestra.run.smithi189.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 262 kB/s | 216 kB 00:00 2024-03-16T20:23:43.194 INFO:teuthology.orchestra.run.smithi189.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 184 kB/s | 44 kB 00:00 2024-03-16T20:23:43.286 INFO:teuthology.orchestra.run.smithi189.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-16T20:23:43.312 INFO:teuthology.orchestra.run.smithi189.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 659 kB/s | 94 kB 00:00 2024-03-16T20:23:43.437 INFO:teuthology.orchestra.run.smithi189.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-16T20:23:43.637 INFO:teuthology.orchestra.run.smithi189.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-16T20:23:43.662 INFO:teuthology.orchestra.run.smithi189.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-16T20:23:43.688 INFO:teuthology.orchestra.run.smithi189.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-03-16T20:23:43.738 INFO:teuthology.orchestra.run.smithi189.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-03-16T20:23:43.788 INFO:teuthology.orchestra.run.smithi189.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-03-16T20:23:43.839 INFO:teuthology.orchestra.run.smithi189.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-16T20:23:43.939 INFO:teuthology.orchestra.run.smithi189.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-16T20:23:44.206 INFO:teuthology.orchestra.run.smithi189.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-16T20:23:44.348 INFO:teuthology.orchestra.run.smithi189.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-16T20:23:44.390 INFO:teuthology.orchestra.run.smithi189.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-16T20:23:44.414 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-03-16T20:23:44.432 INFO:teuthology.orchestra.run.smithi189.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 776 kB/s | 32 kB 00:00 2024-03-16T20:23:44.474 INFO:teuthology.orchestra.run.smithi189.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-03-16T20:23:44.532 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T20:23:44.533 INFO:teuthology.orchestra.run.smithi189.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-16T20:23:44.566 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T20:23:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:44.606 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:44.618 INFO:teuthology.orchestra.run.smithi189.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-16T20:23:44.652 INFO:teuthology.orchestra.run.smithi189.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.9 MB/s | 358 kB 00:00 2024-03-16T20:23:44.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T20:23:44.686 INFO:teuthology.orchestra.run.smithi189.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 1.0 MB/s | 35 kB 00:00 2024-03-16T20:23:44.710 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T20:23:44.711 INFO:teuthology.orchestra.run.smithi189.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-16T20:23:44.743 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T20:23:44.744 INFO:teuthology.orchestra.run.smithi189.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-16T20:23:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T20:23:44.804 INFO:teuthology.orchestra.run.smithi189.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-16T20:23:44.820 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T20:23:44.845 INFO:teuthology.orchestra.run.smithi189.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-03-16T20:23:44.887 INFO:teuthology.orchestra.run.smithi189.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 423 kB/s | 18 kB 00:00 2024-03-16T20:23:44.929 INFO:teuthology.orchestra.run.smithi189.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-03-16T20:23:44.996 INFO:teuthology.orchestra.run.smithi189.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.2 MB/s | 2.0 MB 00:00 2024-03-16T20:23:45.021 INFO:teuthology.orchestra.run.smithi189.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-16T20:23:45.047 INFO:teuthology.orchestra.run.smithi189.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-03-16T20:23:45.089 INFO:teuthology.orchestra.run.smithi189.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-16T20:23:45.147 INFO:teuthology.orchestra.run.smithi189.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-16T20:23:45.163 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T20:23:45.189 INFO:teuthology.orchestra.run.smithi189.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-03-16T20:23:45.218 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:45.248 INFO:teuthology.orchestra.run.smithi189.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-16T20:23:45.282 INFO:teuthology.orchestra.run.smithi189.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 3.9 MB/s | 1.0 MB 00:00 2024-03-16T20:23:45.307 INFO:teuthology.orchestra.run.smithi189.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-03-16T20:23:45.324 INFO:teuthology.orchestra.run.smithi189.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-16T20:23:45.334 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:45.349 INFO:teuthology.orchestra.run.smithi189.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 677 kB/s | 28 kB 00:00 2024-03-16T20:23:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:45.374 INFO:teuthology.orchestra.run.smithi189.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-03-16T20:23:45.416 INFO:teuthology.orchestra.run.smithi189.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-16T20:23:45.442 INFO:teuthology.orchestra.run.smithi189.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-16T20:23:45.454 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:45.467 INFO:teuthology.orchestra.run.smithi189.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-03-16T20:23:45.474 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:23:45.492 INFO:teuthology.orchestra.run.smithi189.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-16T20:23:45.517 INFO:teuthology.orchestra.run.smithi189.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-16T20:23:45.542 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T20:23:45.543 INFO:teuthology.orchestra.run.smithi189.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-16T20:23:45.585 INFO:teuthology.orchestra.run.smithi189.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-03-16T20:23:45.610 INFO:teuthology.orchestra.run.smithi189.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-03-16T20:23:45.652 INFO:teuthology.orchestra.run.smithi189.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-16T20:23:45.717 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T20:23:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T20:23:45.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:23:45.811 INFO:teuthology.orchestra.run.smithi189.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-16T20:23:45.905 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:23:45.934 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:23:45.994 INFO:teuthology.orchestra.run.smithi189.stdout:(150/174): librados2-19.0.0-1681.gaba3c1d2.el9. 9.9 MB/s | 3.4 MB 00:00 2024-03-16T20:23:45.998 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:23:46.036 INFO:teuthology.orchestra.run.smithi189.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-16T20:23:46.046 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T20:23:46.068 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:23:46.085 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:23:46.128 INFO:teuthology.orchestra.run.smithi189.stdout:(152/174): librbd1-19.0.0-1681.gaba3c1d2.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-03-16T20:23:46.212 INFO:teuthology.orchestra.run.smithi189.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-16T20:23:46.236 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:23:46.262 INFO:teuthology.orchestra.run.smithi189.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 852 kB/s | 42 kB 00:00 2024-03-16T20:23:46.266 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:23:46.298 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T20:23:46.312 INFO:teuthology.orchestra.run.smithi189.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 9.9 MB/s | 1.8 MB 00:00 2024-03-16T20:23:46.371 INFO:teuthology.orchestra.run.smithi189.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-16T20:23:46.396 INFO:teuthology.orchestra.run.smithi189.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.8 MB/s | 248 kB 00:00 2024-03-16T20:23:46.399 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T20:23:46.430 INFO:teuthology.orchestra.run.smithi189.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 622 kB/s | 36 kB 00:00 2024-03-16T20:23:46.480 INFO:teuthology.orchestra.run.smithi189.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-16T20:23:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:23:46.522 INFO:teuthology.orchestra.run.smithi189.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-16T20:23:46.544 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:23:46.547 INFO:teuthology.orchestra.run.smithi189.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-16T20:23:46.606 INFO:teuthology.orchestra.run.smithi189.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-16T20:23:46.612 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:46.640 INFO:teuthology.orchestra.run.smithi189.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-16T20:23:46.695 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:46.698 INFO:teuthology.orchestra.run.smithi189.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-16T20:23:46.723 INFO:teuthology.orchestra.run.smithi189.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 765 kB/s | 64 kB 00:00 2024-03-16T20:23:46.766 INFO:teuthology.orchestra.run.smithi189.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 429 kB/s | 29 kB 00:00 2024-03-16T20:23:46.866 INFO:teuthology.orchestra.run.smithi189.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-16T20:23:46.922 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:46.941 INFO:teuthology.orchestra.run.smithi189.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-16T20:23:47.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T20:23:47.050 INFO:teuthology.orchestra.run.smithi189.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-16T20:23:47.092 INFO:teuthology.orchestra.run.smithi189.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-03-16T20:23:47.134 INFO:teuthology.orchestra.run.smithi189.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 373 kB/s | 31 kB 00:00 2024-03-16T20:23:47.209 INFO:teuthology.orchestra.run.smithi189.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-16T20:23:47.259 INFO:teuthology.orchestra.run.smithi189.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-16T20:23:47.301 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T20:23:47.358 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T20:23:47.385 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T20:23:47.429 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T20:23:47.507 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T20:23:47.544 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-16T20:23:47.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T20:23:47.604 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T20:23:47.638 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T20:23:47.672 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T20:23:47.682 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T20:23:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:47.719 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T20:23:47.729 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:47.763 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T20:23:47.795 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T20:23:47.800 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T20:23:47.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T20:23:47.857 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T20:23:47.896 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T20:23:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T20:23:47.928 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 39/198 2024-03-16T20:23:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T20:23:47.972 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 40/198 2024-03-16T20:23:47.999 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 41/198 2024-03-16T20:23:48.073 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T20:23:48.123 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T20:23:48.129 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T20:23:48.202 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T20:23:48.347 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T20:23:48.402 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T20:23:48.505 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T20:23:48.551 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:48.681 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:48.711 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:48.803 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:48.840 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:23:48.887 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T20:23:48.960 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T20:23:49.040 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T20:23:49.078 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T20:23:49.122 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T20:23:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T20:23:49.149 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:23:49.181 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:23:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:23:49.221 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T20:23:49.255 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:23:49.271 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T20:23:49.283 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:23:49.309 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 53/198 2024-03-16T20:23:49.338 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T20:23:49.348 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:23:49.396 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T20:23:49.418 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:23:49.434 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:23:49.476 INFO:teuthology.orchestra.run.smithi189.stdout:(174/174): openssl-devel-3.0.7-27.el9.x86_64.rp 518 kB/s | 4.1 MB 00:08 2024-03-16T20:23:49.481 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-16T20:23:49.481 INFO:teuthology.orchestra.run.smithi189.stdout:Total 13 MB/s | 224 MB 00:17 2024-03-16T20:23:49.566 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T20:23:49.609 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:23:49.644 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:23:49.691 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T20:23:49.893 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T20:23:49.911 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T20:23:49.942 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-16T20:23:49.983 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T20:23:50.026 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:23:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T20:23:50.168 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T20:23:50.215 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-16T20:23:50.215 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-16T20:23:50.229 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T20:23:50.238 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:23:50.285 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T20:23:50.360 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T20:23:50.374 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:50.395 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T20:23:50.432 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T20:23:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T20:23:50.515 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T20:23:50.554 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:50.559 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T20:23:50.599 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T20:23:50.752 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T20:23:50.759 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:23:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T20:23:50.834 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T20:23:50.871 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T20:23:50.876 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T20:23:50.975 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T20:23:50.976 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T20:23:51.008 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T20:23:51.009 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T20:23:51.035 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T20:23:51.049 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T20:23:51.072 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T20:23:51.082 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T20:23:51.107 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T20:23:51.149 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T20:23:51.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T20:23:51.200 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T20:23:51.246 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T20:23:51.313 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T20:23:51.314 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T20:23:51.353 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T20:23:51.397 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T20:23:51.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T20:23:51.430 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T20:23:51.457 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T20:23:51.466 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T20:23:51.523 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T20:23:51.528 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T20:23:51.562 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 39/198 2024-03-16T20:23:51.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T20:23:51.606 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 40/198 2024-03-16T20:23:51.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 41/198 2024-03-16T20:23:51.730 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T20:23:51.774 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T20:23:51.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T20:23:51.828 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T20:23:51.850 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T20:23:51.976 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T20:23:51.988 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T20:23:52.008 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-16T20:23:52.008 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-16T20:23:52.017 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T20:23:52.035 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T20:23:52.049 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T20:23:52.080 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T20:23:52.108 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T20:23:52.143 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T20:23:52.182 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T20:23:52.222 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T20:23:52.251 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T20:23:52.332 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T20:23:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T20:23:52.406 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T20:23:52.445 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T20:23:52.479 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T20:23:52.565 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T20:23:52.592 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T20:23:52.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T20:23:52.675 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T20:23:52.702 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T20:23:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T20:23:52.772 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T20:23:52.782 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T20:23:52.840 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:23:52.850 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:23:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T20:23:52.894 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T20:23:52.930 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T20:23:52.945 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T20:23:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 53/198 2024-03-16T20:23:53.000 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T20:23:53.267 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T20:23:53.449 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T20:23:53.543 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T20:23:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T20:23:53.735 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T20:23:53.739 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T20:23:53.781 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T20:23:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T20:23:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T20:23:53.846 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T20:23:53.865 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T20:23:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T20:23:53.949 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T20:23:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T20:23:53.985 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T20:23:53.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T20:23:54.033 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T20:23:54.068 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T20:23:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T20:23:54.154 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T20:23:54.167 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T20:23:54.197 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T20:23:54.277 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T20:23:54.294 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T20:23:54.350 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T20:23:54.381 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T20:23:54.415 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T20:23:54.513 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T20:23:54.547 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T20:23:54.587 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T20:23:54.612 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T20:23:54.637 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T20:23:54.744 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T20:23:54.876 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T20:23:54.964 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T20:23:54.985 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T20:23:55.029 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T20:23:55.074 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T20:23:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T20:23:55.151 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T20:23:55.345 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T20:23:55.400 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T20:23:55.542 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T20:23:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T20:23:55.621 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T20:23:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T20:23:55.689 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T20:23:55.723 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T20:23:55.762 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T20:23:55.794 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T20:23:55.832 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T20:23:55.902 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T20:23:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T20:23:55.976 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T20:23:56.018 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T20:23:56.052 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T20:23:56.144 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T20:23:56.235 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T20:23:56.283 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T20:23:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T20:23:56.371 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T20:23:56.385 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T20:23:56.463 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T20:23:56.479 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T20:23:56.485 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T20:23:56.517 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T20:23:56.533 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T20:23:56.544 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T20:23:56.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T20:23:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:23:56.990 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:23:57.020 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 125/198 2024-03-16T20:23:57.056 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T20:23:57.303 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T20:23:57.354 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T20:23:57.388 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T20:23:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T20:23:57.527 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T20:23:57.536 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T20:23:57.568 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:23:57.572 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T20:23:57.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:23:57.653 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 129/198 2024-03-16T20:23:57.758 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T20:23:57.794 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T20:23:57.875 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T20:23:58.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T20:23:58.678 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T20:23:58.830 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-16T20:23:58.939 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-16T20:23:58.964 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-16T20:23:58.995 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:59.004 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-16T20:23:59.044 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-16T20:23:59.097 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-16T20:23:59.132 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-16T20:23:59.157 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-16T20:23:59.207 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-16T20:23:59.532 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-16T20:23:59.584 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:59.705 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:23:59.743 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:59.827 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:23:59.857 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:23:59.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-16T20:23:59.897 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:23:59.933 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:23:59.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T20:24:00.053 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T20:24:00.072 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-16T20:24:00.084 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T20:24:00.115 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T20:24:00.130 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-16T20:24:00.142 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T20:24:00.158 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:24:00.260 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:24:00.284 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:24:00.339 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 17/198 2024-03-16T20:24:00.390 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-16T20:24:00.402 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:24:00.418 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-16T20:24:00.468 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T20:24:00.498 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:24:00.500 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:00.536 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:00.589 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:24:00.602 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:00.602 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T20:24:00.602 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T20:24:00.602 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:00.618 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:24:00.619 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 125/198 2024-03-16T20:24:00.628 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:00.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-16T20:24:00.656 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T20:24:00.690 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-16T20:24:00.787 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-16T20:24:00.900 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:24:00.961 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-16T20:24:01.030 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:24:01.095 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:24:01.139 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T20:24:01.167 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:24:01.238 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:24:01.269 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 129/198 2024-03-16T20:24:01.305 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-16T20:24:01.417 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-16T20:24:01.512 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-16T20:24:01.541 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-16T20:24:01.568 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-16T20:24:01.604 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-16T20:24:01.677 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-16T20:24:01.731 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-16T20:24:01.770 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-16T20:24:01.835 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-16T20:24:01.877 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-16T20:24:01.921 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-16T20:24:01.953 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-16T20:24:01.981 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-16T20:24:02.051 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-16T20:24:02.086 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 39/198 2024-03-16T20:24:02.121 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 40/198 2024-03-16T20:24:02.150 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 41/198 2024-03-16T20:24:02.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-16T20:24:02.264 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-16T20:24:02.337 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-16T20:24:02.471 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-16T20:24:02.524 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-16T20:24:03.046 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-16T20:24:03.120 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-16T20:24:03.191 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-16T20:24:03.240 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:24:03.247 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-16T20:24:03.280 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-16T20:24:03.312 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-16T20:24:03.350 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 53/198 2024-03-16T20:24:03.379 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-16T20:24:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:03.628 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-16T20:24:03.632 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-16T20:24:03.848 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-16T20:24:03.892 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-16T20:24:04.031 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-16T20:24:04.078 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-16T20:24:04.132 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-16T20:24:04.199 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:04.205 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-16T20:24:04.235 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:04.237 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-16T20:24:04.273 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-16T20:24:04.320 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-16T20:24:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:04.325 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T20:24:04.325 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T20:24:04.325 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-16T20:24:04.369 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:04.587 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-16T20:24:04.625 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-16T20:24:04.759 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-16T20:24:04.820 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-16T20:24:04.851 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-16T20:24:04.900 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-16T20:24:04.994 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-16T20:24:05.034 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-16T20:24:05.073 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-16T20:24:05.108 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-16T20:24:05.132 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-16T20:24:05.203 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-16T20:24:05.332 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-16T20:24:05.432 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-16T20:24:05.474 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-16T20:24:05.522 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-16T20:24:05.577 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-16T20:24:05.761 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-16T20:24:05.802 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-16T20:24:05.929 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-16T20:24:05.967 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-16T20:24:05.999 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-16T20:24:06.030 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-16T20:24:06.058 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-16T20:24:06.092 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-16T20:24:06.131 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-16T20:24:06.163 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-16T20:24:06.193 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-16T20:24:06.269 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-16T20:24:06.303 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-16T20:24:06.328 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-16T20:24:06.369 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-16T20:24:06.396 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-16T20:24:06.503 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-16T20:24:06.593 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-16T20:24:06.635 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-16T20:24:06.666 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-16T20:24:06.706 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-16T20:24:06.803 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-16T20:24:06.860 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-16T20:24:07.583 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-16T20:24:07.631 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-16T20:24:07.665 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-16T20:24:07.705 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-16T20:24:07.794 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-16T20:24:07.841 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-16T20:24:08.019 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-16T20:24:08.047 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-16T20:24:08.124 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-16T20:24:08.766 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-16T20:24:08.878 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-16T20:24:10.088 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-16T20:24:10.163 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-16T20:24:10.194 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-16T20:24:10.225 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-16T20:24:10.253 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-16T20:24:10.560 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-16T20:24:10.600 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:24:10.673 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 124/198 2024-03-16T20:24:10.697 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 125/198 2024-03-16T20:24:10.732 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-16T20:24:11.185 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-16T20:24:11.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:24:11.273 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 128/198 2024-03-16T20:24:11.296 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 129/198 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-03-16T20:24:12.643 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:12.815 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:12.832 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:12.832 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:12.833 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T20:24:12.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:13.531 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:13.567 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:14.102 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 130/198 2024-03-16T20:24:14.136 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:14.218 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 131/198 2024-03-16T20:24:14.218 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-16T20:24:14.218 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-16T20:24:14.218 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:14.246 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:14.361 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:14.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:14.620 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:14.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:15.104 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:15.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:15.230 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T20:24:15.635 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:15.662 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:15.687 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 139/198 2024-03-16T20:24:15.751 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 140/198 2024-03-16T20:24:15.777 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:16.022 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:16.053 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /root 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-16T20:24:16.896 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:17.065 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:17.065 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:17.081 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:17.081 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:17.081 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T20:24:17.081 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:17.094 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:17.201 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:17.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 144/198 2024-03-16T20:24:17.388 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:17.412 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:17.412 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:17.413 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T20:24:17.413 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:17.413 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:17.413 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:17.455 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:17.467 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 147/198 2024-03-16T20:24:18.405 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T20:24:18.432 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:18.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:18.500 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:18.523 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:18.523 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.523 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T20:24:18.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:18.567 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:18.591 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:18.610 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:18.635 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:18.861 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:18.883 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:18.942 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:19.353 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:19.375 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:19.471 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T20:24:19.856 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:19.887 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:19.910 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 139/198 2024-03-16T20:24:19.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 140/198 2024-03-16T20:24:20.001 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:20.246 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:20.277 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:21.299 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:21.327 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:21.432 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:21.597 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 144/198 2024-03-16T20:24:21.629 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:21.652 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:21.705 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:21.716 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:21.770 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 147/198 2024-03-16T20:24:22.673 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T20:24:22.715 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:22.738 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:22.791 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:22.813 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:22.813 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:22.813 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T20:24:22.813 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:22.867 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:22.888 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:22.889 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:22.889 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T20:24:22.889 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:22.889 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:22.889 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:23.218 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:23.239 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:23.239 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:23.240 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T20:24:23.240 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:23.240 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:23.240 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 154/198 2024-03-16T20:24:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 155/198 2024-03-16T20:24:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T20:24:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 157/198 2024-03-16T20:24:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T20:24:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 159/198 2024-03-16T20:24:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T20:24:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T20:24:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T20:24:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 163/198 2024-03-16T20:24:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T20:24:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T20:24:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T20:24:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T20:24:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T20:24:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:25.676 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 132/198 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-03-16T20:24:25.677 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:25.833 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:25.847 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 133/198 2024-03-16T20:24:25.848 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:25.848 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-16T20:24:25.848 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 170/198 2024-03-16T20:24:26.021 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T20:24:26.063 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T20:24:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T20:24:26.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T20:24:26.208 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T20:24:26.233 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T20:24:26.259 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T20:24:26.260 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:26.331 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:26.369 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T20:24:26.369 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:26.453 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T20:24:26.513 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T20:24:26.538 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T20:24:26.563 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T20:24:26.597 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T20:24:26.635 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T20:24:26.667 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T20:24:26.693 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T20:24:26.718 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T20:24:26.743 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:26.769 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T20:24:26.775 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:26.776 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:26.991 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:26.992 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:27.080 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:27.129 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T20:24:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T20:24:27.189 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T20:24:27.256 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:27.294 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 134/198 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-16T20:24:27.319 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:27.536 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:27.558 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 135/198 2024-03-16T20:24:27.558 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:27.558 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-16T20:24:27.559 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:27.559 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-16T20:24:27.559 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:27.930 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:27.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 154/198 2024-03-16T20:24:27.995 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:28.002 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 155/198 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 136/198 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-16T20:24:28.017 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:28.029 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T20:24:28.105 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-16T20:24:28.235 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 157/198 2024-03-16T20:24:28.357 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T20:24:28.411 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 159/198 2024-03-16T20:24:28.465 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T20:24:28.506 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:28.518 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T20:24:28.536 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 138/198 2024-03-16T20:24:28.561 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T20:24:28.568 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 139/198 2024-03-16T20:24:28.594 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 163/198 2024-03-16T20:24:28.628 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 140/198 2024-03-16T20:24:28.635 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T20:24:28.651 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:28.665 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T20:24:28.734 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T20:24:28.797 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T20:24:28.838 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T20:24:28.884 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 141/198 2024-03-16T20:24:28.919 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:29.586 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:29.619 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:29.879 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 142/198 2024-03-16T20:24:29.911 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:30.010 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 143/198 2024-03-16T20:24:30.173 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 144/198 2024-03-16T20:24:30.204 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 145/198 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-16T20:24:30.228 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:30.271 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:30.282 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 146/198 2024-03-16T20:24:30.327 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 147/198 2024-03-16T20:24:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 170/198 2024-03-16T20:24:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T20:24:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T20:24:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T20:24:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T20:24:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T20:24:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T20:24:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T20:24:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T20:24:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T20:24:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T20:24:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T20:24:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T20:24:31.185 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-16T20:24:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T20:24:31.224 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 149/198 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-16T20:24:31.246 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T20:24:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T20:24:31.291 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T20:24:31.313 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 150/198 2024-03-16T20:24:31.313 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:31.313 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-16T20:24:31.313 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T20:24:31.358 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:31.380 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:31.380 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:31.380 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-16T20:24:31.380 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:31.380 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-16T20:24:31.381 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T20:24:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:31.692 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:31.715 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 152/198 2024-03-16T20:24:31.715 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-16T20:24:31.715 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-16T20:24:31.715 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:31.716 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-16T20:24:31.716 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T20:24:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T20:24:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T20:24:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:31.949 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 1/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 3/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 4/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 5/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 6/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 7/198 2024-03-16T20:24:31.950 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 8/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 9/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 13/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 14/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 15/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 17/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 18/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 19/198 2024-03-16T20:24:31.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 20/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 21/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 22/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 23/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 24/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 25/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 26/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 27/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 28/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 29/198 2024-03-16T20:24:31.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 30/198 2024-03-16T20:24:31.969 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 31/198 2024-03-16T20:24:31.969 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 32/198 2024-03-16T20:24:31.969 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 33/198 2024-03-16T20:24:31.969 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 34/198 2024-03-16T20:24:31.969 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 35/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 36/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 37/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 38/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 39/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T20:24:31.970 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T20:24:31.971 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T20:24:31.972 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T20:24:31.973 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T20:24:31.974 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T20:24:31.975 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T20:24:31.976 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T20:24:31.977 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T20:24:31.978 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T20:24:31.979 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T20:24:31.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T20:24:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T20:24:31.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:35.066 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T20:24:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-16.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T20:24:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T20:24:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T20:24:35.073 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T20:24:35.074 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T20:24:35.075 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T20:24:35.076 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T20:24:35.077 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T20:24:35.078 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T20:24:35.079 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T20:24:35.080 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T20:24:35.081 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T20:24:35.081 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-03-16T20:24:35.081 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-03-16T20:24:35.500 DEBUG:teuthology.parallel:result is None 2024-03-16T20:24:36.129 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:36.170 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 154/198 2024-03-16T20:24:36.201 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 155/198 2024-03-16T20:24:36.228 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-16T20:24:36.407 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 157/198 2024-03-16T20:24:36.485 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:36.485 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 1/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 3/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 4/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 5/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 6/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 7/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 8/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 9/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 13/198 2024-03-16T20:24:36.486 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 14/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 15/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 17/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 18/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 19/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 20/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 21/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 22/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 23/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 24/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 25/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 26/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 27/198 2024-03-16T20:24:36.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 28/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 29/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 30/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 31/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 32/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 33/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 34/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 35/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 36/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 37/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 38/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 39/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T20:24:36.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T20:24:36.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T20:24:36.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T20:24:36.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T20:24:36.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T20:24:36.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T20:24:36.494 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T20:24:36.495 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T20:24:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T20:24:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T20:24:36.499 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T20:24:36.500 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:36.500 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T20:24:36.500 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T20:24:36.500 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T20:24:36.521 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-16T20:24:36.560 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 159/198 2024-03-16T20:24:36.598 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-16T20:24:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-16T20:24:36.696 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-16T20:24:36.712 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 163/198 2024-03-16T20:24:36.745 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-16T20:24:36.774 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-16T20:24:36.836 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-16T20:24:36.897 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-16T20:24:36.930 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-16T20:24:37.509 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:37.545 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:37.564 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-16T20:24:38.383 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 170/198 2024-03-16T20:24:38.465 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-16T20:24:38.505 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-16T20:24:38.586 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-16T20:24:38.635 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-16T20:24:38.660 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-16T20:24:38.702 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-16T20:24:38.728 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-16T20:24:38.728 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:38.816 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-16T20:24:38.846 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-16T20:24:38.846 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:38.931 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-16T20:24:38.957 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-16T20:24:38.990 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-16T20:24:39.015 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-16T20:24:39.040 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-16T20:24:39.073 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-16T20:24:39.113 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-16T20:24:39.153 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:39.172 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.173 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T20:24:39.174 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T20:24:39.176 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-16.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T20:24:39.177 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T20:24:39.178 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T20:24:39.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T20:24:39.180 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T20:24:39.181 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T20:24:39.182 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T20:24:39.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:39.184 INFO:teuthology.orchestra.run.smithi023.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T20:24:39.185 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T20:24:39.185 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T20:24:39.185 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T20:24:39.185 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T20:24:39.185 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T20:24:39.186 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-16T20:24:39.186 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T20:24:39.186 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:24:39.186 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-16T20:24:39.204 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-16T20:24:39.229 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:39.254 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-16T20:24:39.260 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:39.260 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:39.440 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-16T20:24:39.440 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:39.532 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-16T20:24:39.556 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:39.581 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-16T20:24:39.614 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-16T20:24:39.641 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-16T20:24:39.697 DEBUG:teuthology.parallel:result is None 2024-03-16T20:24:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:44.567 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-16T20:24:44.567 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 1/198 2024-03-16T20:24:44.567 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 3/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 4/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1681.gaba3c 5/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 6/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 7/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 8/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 9/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 10/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 11/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 12/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 13/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 14/198 2024-03-16T20:24:44.568 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_ 15/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 16/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 17/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_6 18/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86 19/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 20/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el 21/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9. 22/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_6 23/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 24/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 25/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 26/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 27/198 2024-03-16T20:24:44.569 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 28/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 29/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2. 30/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noa 31/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.n 32/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba 33/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el 34/198 2024-03-16T20:24:44.586 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 35/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.e 36/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 37/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 38/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 39/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-16T20:24:44.587 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-16T20:24:44.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-16T20:24:44.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-16T20:24:44.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-16T20:24:44.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-16T20:24:44.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-16T20:24:44.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-16T20:24:44.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 151/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-16T20:24:44.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 153/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-16T20:24:44.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-16T20:24:44.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-16T20:24:44.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-16T20:24:44.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-16T20:24:44.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-16T20:24:44.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-16T20:24:44.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-16T20:24:47.532 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: acl-2.3.1-4.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-16T20:24:47.533 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.534 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.535 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.536 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-1681.gaba3c1d2.el9.noarch 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-16T20:24:47.537 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: jq-1.6-16.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-16T20:24:47.538 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-16T20:24:47.539 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.11-1.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-16T20:24:47.540 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-16T20:24:47.541 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-16T20:24:47.542 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-16T20:24:47.543 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-16T20:24:47.544 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-1681.gaba3c1d2.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-16T20:24:47.545 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-16T20:24:47.546 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-16T20:24:47.546 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:24:47.546 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-16T20:24:48.044 DEBUG:teuthology.parallel:result is None 2024-03-16T20:24:48.045 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:48.187 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T20:24:48.212 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.213 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.213 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681.gaba3c1d2 is installed. 2024-03-16T20:24:48.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:48.550 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T20:24:48.576 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.576 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.576 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681.gaba3c1d2 is installed. 2024-03-16T20:24:48.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:48.719 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-16T20:24:48.744 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.744 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1681.gaba3c1d2.el9 2024-03-16T20:24:48.745 INFO:teuthology.task.install:The correct ceph version 19.0.0-1681.gaba3c1d2 is installed. 2024-03-16T20:24:48.746 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-16T20:24:48.746 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:24:48.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T20:24:48.781 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:24:48.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T20:24:48.820 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:24:48.821 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-16T20:24:48.855 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-16T20:24:48.855 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:24:48.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T20:24:48.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T20:24:48.953 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:24:48.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T20:24:48.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T20:24:49.061 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:24:49.062 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-03-16T20:24:49.093 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-16T20:24:49.164 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-16T20:24:49.164 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:24:49.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T20:24:49.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T20:24:49.258 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:24:49.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T20:24:49.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T20:24:49.360 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:24:49.360 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-16T20:24:49.391 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-16T20:24:49.461 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-16T20:24:49.461 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:24:49.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T20:24:49.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T20:24:49.559 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:24:49.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T20:24:49.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T20:24:49.660 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:24:49.660 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-03-16T20:24:49.691 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-16T20:24:49.761 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T20:24:49.847 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'SLOW_OPS', 'slow request'], 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-16T20:24:49.847 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:49.847 INFO:tasks.cephadm:Cluster fsid is 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:24:49.848 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T20:24:49.848 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.31', 'mon.c': '172.21.15.189'} 2024-03-16T20:24:49.848 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-16T20:24:49.848 INFO:tasks.cephadm:First mgr is x 2024-03-16T20:24:49.848 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T20:24:49.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-16T20:24:49.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-03-16T20:24:49.906 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-03-16T20:24:49.936 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T20:24:49.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:50.074 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77659/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1681-gaba3c1d2', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1681.gaba3c1d2'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 03:39:08.164055', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-qdb-squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-16T20:24:50.310 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-lusov-qdb-squid, sha1 aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:24:50.312 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/x86_64/flavors/default/cephadm 2024-03-16T20:24:50.312 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:51.452 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 766937 Mar 16 20:24 /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:51.452 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:52.631 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 766937 Mar 16 20:24 /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:52.632 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb-squid/aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:53.763 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 766937 Mar 16 20:24 /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:53.763 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:53.782 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:53.801 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T20:24:53.835 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b on all hosts... 2024-03-16T20:24:53.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b pull 2024-03-16T20:24:53.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b pull 2024-03-16T20:24:53.844 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b pull 2024-03-16T20:24:54.677 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b... 2024-03-16T20:24:54.749 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b... 2024-03-16T20:24:54.853 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b... 2024-03-16T20:25:28.313 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:25:28.313 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev)", 2024-03-16T20:25:28.313 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0", 2024-03-16T20:25:28.314 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-16T20:25:28.314 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3" 2024-03-16T20:25:28.314 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:25:28.314 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:25:37.298 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-16T20:25:37.298 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev)", 2024-03-16T20:25:37.298 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0", 2024-03-16T20:25:37.299 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-03-16T20:25:37.299 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3" 2024-03-16T20:25:37.299 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-03-16T20:25:37.299 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-16T20:25:37.496 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-16T20:25:37.496 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev)", 2024-03-16T20:25:37.496 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0", 2024-03-16T20:25:37.497 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-03-16T20:25:37.497 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3" 2024-03-16T20:25:37.497 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-03-16T20:25:37.497 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-16T20:25:37.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-16T20:25:37.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-03-16T20:25:37.591 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-03-16T20:25:37.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-16T20:25:37.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-03-16T20:25:37.683 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-03-16T20:25:37.710 INFO:tasks.cephadm:Writing seed config... 2024-03-16T20:25:37.711 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-16T20:25:37.711 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-16T20:25:37.711 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-16T20:25:37.711 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-16T20:25:37.712 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-16T20:25:37.713 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-16T20:25:37.714 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-16T20:25:37.715 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-16T20:25:37.715 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-16T20:25:37.715 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-16T20:25:37.715 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T20:25:37.715 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-16T20:25:37.716 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:25:37.716 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T20:25:37.733 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3d249ea8-e3d3-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-16T20:25:37.733 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T20:25:37.776 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T20:25:37.818 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T20:25:37.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b -v bootstrap --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:25:38.017 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-16T20:25:38.017 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b', '-v', 'bootstrap', '--fsid', '3d249ea8-e3d3-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-16T20:25:38.041 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T20:25:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T20:25:38.042 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-16T20:25:38.065 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T20:25:38.065 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-16T20:25:38.065 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-16T20:25:38.073 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T20:25:38.073 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T20:25:38.079 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T20:25:38.079 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:25:38.085 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-16T20:25:38.091 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-16T20:25:38.091 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-16T20:25:38.091 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-16T20:25:38.114 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T20:25:38.114 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T20:25:38.114 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-16T20:25:38.114 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-16T20:25:38.120 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T20:25:38.120 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T20:25:38.125 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T20:25:38.125 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:25:38.132 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 140390772654032 on /run/cephadm/3d249ea8-e3d3-11ee-95c9-87774f69a715.lock 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 140390772654032 acquired on /run/cephadm/3d249ea8-e3d3-11ee-95c9-87774f69a715.lock 2024-03-16T20:25:38.138 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-16T20:25:38.139 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-16T20:25:38.139 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-16T20:25:38.142 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2024-03-16T20:25:38.142 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-16T20:25:38.144 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T20:25:38.144 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-16T20:25:38.144 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-16T20:25:38.147 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T20:25:38.148 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-16T20:25:38.148 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-16T20:25:38.148 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T20:25:38.148 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T20:25:38.149 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b... 2024-03-16T20:25:38.639 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 2024-03-16T20:25:38.639 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b... 2024-03-16T20:25:38.639 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T20:25:38.639 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:70a48e3dafcdedd8abcbc488c0076bda17d489568a5d72d4513d3d6eecb1880d 2024-03-16T20:25:38.639 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T20:25:38.640 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 2024-03-16T20:25:38.640 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T20:25:39.493 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev) 2024-03-16T20:25:39.493 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev) 2024-03-16T20:25:39.493 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T20:25:40.349 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-16T20:25:40.349 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-16T20:25:41.321 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBEAPZlcBCaJRAAnTMPRc98+g5cW4CBzUNNgg== 2024-03-16T20:25:42.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBFAPZlVcf7JRAAVucvq7HpdZwgliMs/NddRA== 2024-03-16T20:25:43.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBGAPZlDGdYGBAALfVUAKqblrSPENPgvtvbwA== 2024-03-16T20:25:43.115 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:43.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T20:25:43.955 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:25:43.955 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-16T20:25:44.884 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-16T20:25:45.089 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-16T20:25:45.325 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T20:25:45.504 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target. 2024-03-16T20:25:45.504 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target. 2024-03-16T20:25:45.760 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a 2024-03-16T20:25:45.760 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T20:25:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target.wants/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@.service. 2024-03-16T20:25:46.499 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T20:25:46.499 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-16T20:25:46.506 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T20:25:46.506 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:25:46.507 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-16T20:25:46.507 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T20:25:46.507 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-16T20:25:46.507 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.502104s) 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-16T20:25:47.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78dd189700 1 Processor -- start 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78dd189700 1 -- start start 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78dd189700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78dd189700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d807bdd0 con 0x7f78d8108610 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:47.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54282/0 (socket says 172.21.15.23:54282) 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.910+0000 7f78d6d9d700 1 -- 172.21.15.23:0/2121576140 learned_addr learned my addr 172.21.15.23:0/2121576140 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.911+0000 7f78d6d9d700 1 -- 172.21.15.23:0/2121576140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d810af40 con 0x7f78d8108610 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.911+0000 7f78d6d9d700 1 --2- 172.21.15.23:0/2121576140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f78c80098a0 tx=0x7f78c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ddef5baba4b7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.911+0000 7f78d5d9b700 1 -- 172.21.15.23:0/2121576140 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78c8004160 con 0x7f78d8108610 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.911+0000 7f78d5d9b700 1 -- 172.21.15.23:0/2121576140 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c80042c0 con 0x7f78d8108610 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.911+0000 7f78d5d9b700 1 -- 172.21.15.23:0/2121576140 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78c80045a0 con 0x7f78d8108610 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 -- 172.21.15.23:0/2121576140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 msgr2=0x7f78d810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 --2- 172.21.15.23:0/2121576140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f78c80098a0 tx=0x7f78c8009bb0 comp rx=0 tx=0).stop 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 -- 172.21.15.23:0/2121576140 shutdown_connections 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 --2- 172.21.15.23:0/2121576140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d810aa00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 -- 172.21.15.23:0/2121576140 >> 172.21.15.23:0/2121576140 conn(0x7f78d8102020 msgr2=0x7f78d8104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:47.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 -- 172.21.15.23:0/2121576140 shutdown_connections 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 -- 172.21.15.23:0/2121576140 wait complete. 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.912+0000 7f78dd189700 1 Processor -- start 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78dd189700 1 -- start start 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78dd189700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78dd189700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d819b510 con 0x7f78d8108610 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54292/0 (socket says 172.21.15.23:54292) 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78d6d9d700 1 -- 172.21.15.23:0/3949427 learned_addr learned my addr 172.21.15.23:0/3949427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78d6d9d700 1 -- 172.21.15.23:0/3949427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009550 con 0x7f78d8108610 2024-03-16T20:25:47.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.913+0000 7f78d6d9d700 1 --2- 172.21.15.23:0/3949427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f78d807b210 tx=0x7f78c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78c8011630 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c8011790 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d819b710 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78c8011a30 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d819bbb0 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f78c8022070 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.914+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f78c801c450 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.915+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d8194e40 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.917+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f78c8011b90 con 0x7f78d8108610 2024-03-16T20:25:47.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.991+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f78d810b400 con 0x7f78d8108610 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.991+0000 7f78bffff700 1 -- 172.21.15.23:0/3949427 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f78c8011d30 con 0x7f78d8108610 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 msgr2=0x7f78d819afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 --2- 172.21.15.23:0/3949427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f78d807b210 tx=0x7f78c8004750 comp rx=0 tx=0).stop 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 shutdown_connections 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 --2- 172.21.15.23:0/3949427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78d8108610 0x7f78d819afd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 >> 172.21.15.23:0/3949427 conn(0x7f78d8102020 msgr2=0x7f78d8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 shutdown_connections 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:46.992+0000 7f78dd189700 1 -- 172.21.15.23:0/3949427 wait complete. 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-16T20:25:47.482 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T20:25:48.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:48.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T20:25:48.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T20:25:48.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.917+0000 7fa28a184700 1 Processor -- start 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.917+0000 7fa28a184700 1 -- start start 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.917+0000 7fa28a184700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.917+0000 7fa28a184700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2841085e0 con 0x7fa284107bf0 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54302/0 (socket says 172.21.15.23:54302) 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2837fe700 1 -- 172.21.15.23:0/380841955 learned_addr learned my addr 172.21.15.23:0/380841955 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2837fe700 1 -- 172.21.15.23:0/380841955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa284108d50 con 0x7fa284107bf0 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2837fe700 1 --2- 172.21.15.23:0/380841955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa2740098a0 tx=0x7fa274009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1c4f5e1cccab4e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2827fc700 1 -- 172.21.15.23:0/380841955 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa274004160 con 0x7fa284107bf0 2024-03-16T20:25:48.460 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2827fc700 1 -- 172.21.15.23:0/380841955 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2740042c0 con 0x7fa284107bf0 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.918+0000 7fa2827fc700 1 -- 172.21.15.23:0/380841955 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2740045a0 con 0x7fa284107bf0 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 -- 172.21.15.23:0/380841955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 msgr2=0x7fa284108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 --2- 172.21.15.23:0/380841955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa2740098a0 tx=0x7fa274009bb0 comp rx=0 tx=0).stop 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 -- 172.21.15.23:0/380841955 shutdown_connections 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 --2- 172.21.15.23:0/380841955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa284108010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 -- 172.21.15.23:0/380841955 >> 172.21.15.23:0/380841955 conn(0x7fa2841031d0 msgr2=0x7fa2841055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 -- 172.21.15.23:0/380841955 shutdown_connections 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.919+0000 7fa28a184700 1 -- 172.21.15.23:0/380841955 wait complete. 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa28a184700 1 Processor -- start 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa28a184700 1 -- start start 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa28a184700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa28a184700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419b4f0 con 0x7fa284107bf0 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa2837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa2837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54308/0 (socket says 172.21.15.23:54308) 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa2837fe700 1 -- 172.21.15.23:0/3392013594 learned_addr learned my addr 172.21.15.23:0/3392013594 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.920+0000 7fa2837fe700 1 -- 172.21.15.23:0/3392013594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274009550 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa2837fe700 1 --2- 172.21.15.23:0/3392013594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa274004000 tx=0x7fa274004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa274011630 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa274011790 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28419b6f0 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa274011a30 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28419bb90 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa274022070 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.921+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa27401c450 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.922+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa284194c80 con 0x7fa284107bf0 2024-03-16T20:25:48.462 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.924+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa274011b90 con 0x7fa284107bf0 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.994+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa2841090c0 con 0x7fa284107bf0 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.998+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7fa274011de0 con 0x7fa284107bf0 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.998+0000 7fa280ff9700 1 -- 172.21.15.23:0/3392013594 <== mon.0 v2:172.21.15.23:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa27401c6b0 con 0x7fa284107bf0 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 msgr2=0x7fa28419afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 --2- 172.21.15.23:0/3392013594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa274004000 tx=0x7fa274004750 comp rx=0 tx=0).stop 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 shutdown_connections 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 --2- 172.21.15.23:0/3392013594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa284107bf0 0x7fa28419afb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 >> 172.21.15.23:0/3392013594 conn(0x7fa2841031d0 msgr2=0x7fa284191dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 shutdown_connections 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:47.999+0000 7fa28a184700 1 -- 172.21.15.23:0/3392013594 wait complete. 2024-03-16T20:25:48.463 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-16T20:25:48.577 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:48 smithi023 ceph-mon[31507]: from='client.? 172.21.15.23:0/3392013594' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T20:25:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc40aa3e700 1 Processor -- start 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc40aa3e700 1 -- start start 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc40aa3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc40aa3e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041085e0 con 0x7fc404107bf0 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc403fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc403fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54310/0 (socket says 172.21.15.23:54310) 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc403fff700 1 -- 172.21.15.23:0/2993818485 learned_addr learned my addr 172.21.15.23:0/2993818485 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc403fff700 1 -- 172.21.15.23:0/2993818485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404108d50 con 0x7fc404107bf0 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.882+0000 7fc403fff700 1 --2- 172.21.15.23:0/2993818485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40098a0 tx=0x7fc3f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec70b0c103099f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc402ffd700 1 -- 172.21.15.23:0/2993818485 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3f4004160 con 0x7fc404107bf0 2024-03-16T20:25:49.506 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc402ffd700 1 -- 172.21.15.23:0/2993818485 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3f40042c0 con 0x7fc404107bf0 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc402ffd700 1 -- 172.21.15.23:0/2993818485 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3f4011ac0 con 0x7fc404107bf0 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 -- 172.21.15.23:0/2993818485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 msgr2=0x7fc404108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 --2- 172.21.15.23:0/2993818485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40098a0 tx=0x7fc3f4009bb0 comp rx=0 tx=0).stop 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 -- 172.21.15.23:0/2993818485 shutdown_connections 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 --2- 172.21.15.23:0/2993818485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc404108010 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 -- 172.21.15.23:0/2993818485 >> 172.21.15.23:0/2993818485 conn(0x7fc4041031d0 msgr2=0x7fc4041055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 -- 172.21.15.23:0/2993818485 shutdown_connections 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.883+0000 7fc40aa3e700 1 -- 172.21.15.23:0/2993818485 wait complete. 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc40aa3e700 1 Processor -- start 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc40aa3e700 1 -- start start 2024-03-16T20:25:49.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc40aa3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc40aa3e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40419be30 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc403fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc403fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54320/0 (socket says 172.21.15.23:54320) 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.884+0000 7fc403fff700 1 -- 172.21.15.23:0/1995978609 learned_addr learned my addr 172.21.15.23:0/1995978609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc403fff700 1 -- 172.21.15.23:0/1995978609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f4009550 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc403fff700 1 --2- 172.21.15.23:0/1995978609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4011780 tx=0x7fc3f4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3f4003af0 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3f4003c50 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40419c030 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3f40197e0 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40419c4d0 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc3f4025460 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.885+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc3f400bb20 con 0x7fc404107bf0 2024-03-16T20:25:49.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.886+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4040690f0 con 0x7fc404107bf0 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.888+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc3f402a060 con 0x7fc404107bf0 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.958+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc40410bf80 con 0x7fc404107bf0 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.958+0000 7fc4017fa700 1 -- 172.21.15.23:0/1995978609 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fc3f4019d60 con 0x7fc404107bf0 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 msgr2=0x7fc40419b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 --2- 172.21.15.23:0/1995978609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4011780 tx=0x7fc3f4011b40 comp rx=0 tx=0).stop 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 shutdown_connections 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 --2- 172.21.15.23:0/1995978609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc404107bf0 0x7fc40419b8f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 >> 172.21.15.23:0/1995978609 conn(0x7fc4041031d0 msgr2=0x7fc404103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 shutdown_connections 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:48.959+0000 7fc40aa3e700 1 -- 172.21.15.23:0/1995978609 wait complete. 2024-03-16T20:25:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-16T20:25:49.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:49 smithi023 ceph-mon[31507]: from='client.? 172.21.15.23:0/1995978609' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:25:49.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:49 smithi023 systemd[1]: Stopping Ceph mon.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:25:50.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:49 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a[31502]: 2024-03-16T20:25:49.781+0000 7f2f0bf0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:25:50.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:49 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a[31502]: 2024-03-16T20:25:49.781+0000 7f2f0bf0c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T20:25:50.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:49 smithi023 podman[31748]: 2024-03-16 20:25:49.96641704 +0000 UTC m=+0.291452327 container died d9f035fece9521a39116e8f9494d6073261c875ef71809759640b7b66bacde3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T20:25:50.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 podman[31748]: 2024-03-16 20:25:50.323628633 +0000 UTC m=+0.648663927 container cleanup d9f035fece9521a39116e8f9494d6073261c875ef71809759640b7b66bacde3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-03-16T20:25:50.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 bash[31748]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a 2024-03-16T20:25:50.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 podman[31762]: 2024-03-16 20:25:50.582908064 +0000 UTC m=+0.616121026 container remove d9f035fece9521a39116e8f9494d6073261c875ef71809759640b7b66bacde3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T20:25:50.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T20:25:50.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 systemd[1]: Stopped Ceph mon.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:25:50.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 systemd[1]: Starting Ceph mon.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:25:51.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:50 smithi023 podman[31860]: 2024-03-16 20:25:50.858842008 +0000 UTC m=+0.017448681 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:25:52.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:51 smithi023 podman[31860]: 2024-03-16 20:25:51.901224654 +0000 UTC m=+1.059831322 container create bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 podman[31860]: 2024-03-16 20:25:53.785829767 +0000 UTC m=+2.944436436 container init bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 podman[31860]: 2024-03-16 20:25:53.789571967 +0000 UTC m=+2.948178648 container start bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev), process ceph-mon, pid 2 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: pidfile_write: ignore empty --pid-file 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: load: jerasure load: lrc 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: RocksDB version: 7.9.2 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Git sha 0 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Compile date 2024-03-15 02:27:09 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: DB SUMMARY 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: DB Session ID: DK0734U6N93H57ZWW0EL 2024-03-16T20:25:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: CURRENT file: CURRENT 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100852 ; 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.error_if_exists: 0 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.create_if_missing: 0 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.paranoid_checks: 1 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.env: 0x55f608e01ae0 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T20:25:54.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.info_log: 0x55f60b04b900 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.statistics: (nil) 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.use_fsync: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_log_file_size: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_fallocate: 1 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.use_direct_reads: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T20:25:54.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.db_log_dir: 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.wal_dir: 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.write_buffer_manager: 0x55f60b01f400 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T20:25:54.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.unordered_write: 0 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.row_cache: None 2024-03-16T20:25:54.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.wal_filter: None 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.two_write_queues: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.wal_compression: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.atomic_flush: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.log_readahead_size: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T20:25:54.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_background_jobs: 2 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_background_compactions: -1 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_subcompactions: 1 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T20:25:54.096 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_open_files: -1 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_background_flushes: -1 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Compression algorithms supported: 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kZSTD supported: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kXpressCompression supported: 0 2024-03-16T20:25:54.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kZlibCompression supported: 1 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kSnappyCompression supported: 1 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kLZ4Compression supported: 1 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: kBZip2Compression supported: 0 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.merge_operator: 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_filter: None 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T20:25:54.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f60b04ba00) 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-16T20:25:54.099 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x55f60b04f090 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-16T20:25:54.100 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T20:25:54.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression: NoCompression 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.num_levels: 7 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T20:25:54.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T20:25:54.103 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T20:25:54.104 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T20:25:54.105 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.table_properties_collectors: 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.inplace_update_support: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.bloom_locality: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.max_successive_merges: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T20:25:54.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.ttl: 2592000 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enable_blob_files: false 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.min_blob_size: 0 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T20:25:54.107 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d62d0f1b-aac0-4ca4-85af-efc7b4ed61f9 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620753821864, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620753822988, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95940, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 93933, "index_size": 289, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 14051, "raw_average_key_size": 50, "raw_value_size": 86282, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710620753, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d62d0f1b-aac0-4ca4-85af-efc7b4ed61f9", "db_session_id": "DK0734U6N93H57ZWW0EL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620753823084, "job": 1, "event": "recovery_finished"} 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f60b140000 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: DB pointer 0x55f60b12a000 2024-03-16T20:25:54.108 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T20:25:54.109 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 95.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 95.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T20:25:54.110 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 25.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 25.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x55f60b04f090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???) e1 preinit fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).mds e1 new map 2024-03-16T20:25:54.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).mds e1 print_map 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:25:54.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: fsmap 2024-03-16T20:25:54.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T20:25:54.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:53 smithi023 ceph-mon[31879]: mgrmap e1: no daemons active 2024-03-16T20:25:54.769 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T20:25:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:54 smithi023 bash[31860]: bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 2024-03-16T20:25:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:54 smithi023 systemd[1]: Started Ceph mon.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:25:59.206 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:25:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/958265605' entity='client.admin' 2024-03-16T20:26:02.437 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.869+0000 7fc9f12c1700 1 Processor -- start 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.869+0000 7fc9f12c1700 1 -- start start 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.869+0000 7fc9f12c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.869+0000 7fc9f12c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1084c0 con 0x7fc9ec107b60 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52144/0 (socket says 172.21.15.23:52144) 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9eaffd700 1 -- 172.21.15.23:0/3590697245 learned_addr learned my addr 172.21.15.23:0/3590697245 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:02.442 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9eaffd700 1 -- 172.21.15.23:0/3590697245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec108600 con 0x7fc9ec107b60 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9eaffd700 1 --2- 172.21.15.23:0/3590697245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0098a0 tx=0x7fc9dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0fa4415195bf2b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9e9ffb700 1 -- 172.21.15.23:0/3590697245 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc004160 con 0x7fc9ec107b60 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9e9ffb700 1 -- 172.21.15.23:0/3590697245 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9dc0042c0 con 0x7fc9ec107b60 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.870+0000 7fc9e9ffb700 1 -- 172.21.15.23:0/3590697245 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc011ac0 con 0x7fc9ec107b60 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 -- 172.21.15.23:0/3590697245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 msgr2=0x7fc9ec107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 --2- 172.21.15.23:0/3590697245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0098a0 tx=0x7fc9dc009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 -- 172.21.15.23:0/3590697245 shutdown_connections 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 --2- 172.21.15.23:0/3590697245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec107f80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 -- 172.21.15.23:0/3590697245 >> 172.21.15.23:0/3590697245 conn(0x7fc9ec1031c0 msgr2=0x7fc9ec1055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 -- 172.21.15.23:0/3590697245 shutdown_connections 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.871+0000 7fc9f12c1700 1 -- 172.21.15.23:0/3590697245 wait complete. 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9f12c1700 1 Processor -- start 2024-03-16T20:26:02.443 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9f12c1700 1 -- start start 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9f12c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9f12c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19bda0 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52150/0 (socket says 172.21.15.23:52150) 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9eaffd700 1 -- 172.21.15.23:0/958265605 learned_addr learned my addr 172.21.15.23:0/958265605 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9eaffd700 1 -- 172.21.15.23:0/958265605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc009550 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.872+0000 7fc9eaffd700 1 --2- 172.21.15.23:0/958265605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc003f70 tx=0x7fc9dc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc0039f0 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9dc003b50 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec19bfa0 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc01a760 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec19c440 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc9dc02d3f0 con 0x7fc9ec107b60 2024-03-16T20:26:02.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.873+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc9dc0199b0 con 0x7fc9ec107b60 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.874+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec063610 con 0x7fc9ec107b60 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.876+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc9dc01ad60 con 0x7fc9ec107b60 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.946+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc9ec109990 con 0x7fc9ec107b60 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.948+0000 7fc9d3fff700 1 -- 172.21.15.23:0/958265605 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc9dc0249f0 con 0x7fc9ec107b60 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 msgr2=0x7fc9ec19b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 --2- 172.21.15.23:0/958265605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc003f70 tx=0x7fc9dc011b40 comp rx=0 tx=0).stop 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 shutdown_connections 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 --2- 172.21.15.23:0/958265605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9ec107b60 0x7fc9ec19b860 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 >> 172.21.15.23:0/958265605 conn(0x7fc9ec1031c0 msgr2=0x7fc9ec103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 shutdown_connections 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:25:57.949+0000 7fc9f12c1700 1 -- 172.21.15.23:0/958265605 wait complete. 2024-03-16T20:26:02.445 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T20:26:02.467 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:26:02.467 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-16T20:26:02.468 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T20:26:02.468 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T20:26:02.723 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x 2024-03-16T20:26:02.723 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-16T20:26:02.868 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715.target.wants/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@.service. 2024-03-16T20:26:03.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:02 smithi023 systemd[1]: Starting Ceph mgr.x for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:26:03.371 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 podman[32106]: 2024-03-16 20:26:03.092725493 +0000 UTC m=+0.111763466 container create 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T20:26:03.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 podman[32106]: 2024-03-16 20:26:02.9988929 +0000 UTC m=+0.017930873 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:26:03.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 podman[32106]: 2024-03-16 20:26:03.286588805 +0000 UTC m=+0.305626781 container init 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-16T20:26:03.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 podman[32106]: 2024-03-16 20:26:03.289026799 +0000 UTC m=+0.308064775 container start 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-16T20:26:03.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 bash[32106]: 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 2024-03-16T20:26:03.384 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T20:26:03.385 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-16T20:26:03.391 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T20:26:03.391 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:26:03.391 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-16T20:26:03.391 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T20:26:03.399 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T20:26:03.399 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-16T20:26:03.406 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T20:26:03.407 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:26:03.407 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-16T20:26:03.407 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T20:26:03.407 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-16T20:26:03.407 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-16T20:26:03.631 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 systemd[1]: Started Ceph mgr.x for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:26:03.940 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:03.846+0000 7f7dc7d32200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T20:26:04.219 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3724496353' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T20:26:04.220 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:03.944+0000 7f7dc7d32200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T20:26:04.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T20:26:04.475 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T20:26:04.476 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T20:26:04.477 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T20:26:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T20:25:46.503749+0000", 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:04.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.856+0000 7fbd67bba700 1 Processor -- start 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.856+0000 7fbd67bba700 1 -- start start 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.856+0000 7fbd67bba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.856+0000 7fbd67bba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd601073e0 con 0x7fbd601069f0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd65956700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd65956700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57710/0 (socket says 172.21.15.23:57710) 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd65956700 1 -- 172.21.15.23:0/3567142857 learned_addr learned my addr 172.21.15.23:0/3567142857 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd65956700 1 -- 172.21.15.23:0/3567142857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60107bf0 con 0x7fbd601069f0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd65956700 1 --2- 172.21.15.23:0/3567142857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000bb40 tx=0x7fbd5000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d33ee0823d29e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd64954700 1 -- 172.21.15.23:0/3567142857 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd5000dbe0 con 0x7fbd601069f0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd64954700 1 -- 172.21.15.23:0/3567142857 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd500044f0 con 0x7fbd601069f0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.857+0000 7fbd64954700 1 -- 172.21.15.23:0/3567142857 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd50010470 con 0x7fbd601069f0 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 -- 172.21.15.23:0/3567142857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 msgr2=0x7fbd60106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 --2- 172.21.15.23:0/3567142857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000bb40 tx=0x7fbd5000be50 comp rx=0 tx=0).stop 2024-03-16T20:26:04.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 -- 172.21.15.23:0/3567142857 shutdown_connections 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 --2- 172.21.15.23:0/3567142857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd60106e10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 -- 172.21.15.23:0/3567142857 >> 172.21.15.23:0/3567142857 conn(0x7fbd60101f70 msgr2=0x7fbd601043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 -- 172.21.15.23:0/3567142857 shutdown_connections 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.858+0000 7fbd67bba700 1 -- 172.21.15.23:0/3567142857 wait complete. 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd67bba700 1 Processor -- start 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd67bba700 1 -- start start 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd67bba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd67bba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6019e080 con 0x7fbd601069f0 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd65956700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd65956700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57718/0 (socket says 172.21.15.23:57718) 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd65956700 1 -- 172.21.15.23:0/3724496353 learned_addr learned my addr 172.21.15.23:0/3724496353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd65956700 1 -- 172.21.15.23:0/3724496353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5000b7f0 con 0x7fbd601069f0 2024-03-16T20:26:04.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.859+0000 7fbd65956700 1 --2- 172.21.15.23:0/3724496353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000b340 tx=0x7fbd50010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd50004290 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd50010b60 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd50021b40 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6019e280 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6019e6c0 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbd5002a3f0 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.860+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbd500196f0 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.861+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd60193280 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.863+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbd5002f060 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.937+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbd600690f0 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.937+0000 7fbd56ffd700 1 -- 172.21.15.23:0/3724496353 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fbd50021db0 con 0x7fbd601069f0 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 msgr2=0x7fbd6019db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:04.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 --2- 172.21.15.23:0/3724496353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000b340 tx=0x7fbd50010520 comp rx=0 tx=0).stop 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 shutdown_connections 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 --2- 172.21.15.23:0/3724496353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd601069f0 0x7fbd6019db40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 >> 172.21.15.23:0/3724496353 conn(0x7fbd60101f70 msgr2=0x7fbd601903c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 shutdown_connections 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:03.939+0000 7fbd67bba700 1 -- 172.21.15.23:0/3724496353 wait complete. 2024-03-16T20:26:04.483 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-16T20:26:04.946 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:04.609+0000 7f7dc7d32200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T20:26:04.946 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:04.702+0000 7f7dc7d32200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T20:26:04.946 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:04.844+0000 7f7dc7d32200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T20:26:05.296 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:04.942+0000 7f7dc7d32200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T20:26:05.296 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.027+0000 7f7dc7d32200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T20:26:05.296 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.292+0000 7f7dc7d32200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T20:26:05.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.404+0000 7f7dc7d32200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:05.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.491+0000 7f7dc7d32200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T20:26:06.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.587+0000 7f7dc7d32200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T20:26:06.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:05.764+0000 7f7dc7d32200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T20:26:06.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:06 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:06.764+0000 7f7dc7d32200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T20:26:07.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:06 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2030885525' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T20:26:07.245 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:06 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:06.902+0000 7f7dc7d32200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T20:26:07.245 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:07 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:07.241+0000 7f7dc7d32200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T20:26:07.534 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:07 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:07.331+0000 7f7dc7d32200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T20:26:07.534 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:07 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:07.531+0000 7f7dc7d32200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:07.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:07.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T20:26:07.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T20:26:07.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T20:26:07.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T20:25:46.503749+0000", 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.859+0000 7f7f10032700 1 Processor -- start 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f10032700 1 -- start start 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f10032700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f10032700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080758a0 con 0x7f7f08106350 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f0ddce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:07.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f0ddce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57722/0 (socket says 172.21.15.23:57722) 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f0ddce700 1 -- 172.21.15.23:0/2378851753 learned_addr learned my addr 172.21.15.23:0/2378851753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f0ddce700 1 -- 172.21.15.23:0/2378851753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08106cb0 con 0x7f7f08106350 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.860+0000 7f7f0ddce700 1 --2- 172.21.15.23:0/2378851753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7f040098a0 tx=0x7f7f04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6962519a462f18db server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f0cdcc700 1 -- 172.21.15.23:0/2378851753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f04004160 con 0x7f7f08106350 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f0cdcc700 1 -- 172.21.15.23:0/2378851753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f040042c0 con 0x7f7f08106350 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f0cdcc700 1 -- 172.21.15.23:0/2378851753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f04011ac0 con 0x7f7f08106350 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 -- 172.21.15.23:0/2378851753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 msgr2=0x7f7f08106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 --2- 172.21.15.23:0/2378851753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7f040098a0 tx=0x7f7f04009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 -- 172.21.15.23:0/2378851753 shutdown_connections 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 --2- 172.21.15.23:0/2378851753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f08106770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 -- 172.21.15.23:0/2378851753 >> 172.21.15.23:0/2378851753 conn(0x7f7f08101fb0 msgr2=0x7f7f081043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 -- 172.21.15.23:0/2378851753 shutdown_connections 2024-03-16T20:26:07.719 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.861+0000 7f7f10032700 1 -- 172.21.15.23:0/2378851753 wait complete. 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f10032700 1 Processor -- start 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f10032700 1 -- start start 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f10032700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f10032700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0819e020 con 0x7f7f08106350 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f0ddce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f0ddce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57734/0 (socket says 172.21.15.23:57734) 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.862+0000 7f7f0ddce700 1 -- 172.21.15.23:0/2030885525 learned_addr learned my addr 172.21.15.23:0/2030885525 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7f0ddce700 1 -- 172.21.15.23:0/2030885525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f04009550 con 0x7f7f08106350 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7f0ddce700 1 --2- 172.21.15.23:0/2030885525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009fb0 tx=0x7f7f04011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f040038b0 con 0x7f7f08106350 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f04003a10 con 0x7f7f08106350 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0819e220 con 0x7f7f08106350 2024-03-16T20:26:07.720 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f040195e0 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0819e6c0 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7f04019c50 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.863+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f040249e0 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.864+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08197770 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.866+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7f04003b80 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.940+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7f08107df0 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.941+0000 7f7efeffd700 1 -- 172.21.15.23:0/2030885525 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f7f0401ab00 con 0x7f7f08106350 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.941+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 msgr2=0x7f7f0819dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.941+0000 7f7f10032700 1 --2- 172.21.15.23:0/2030885525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009fb0 tx=0x7f7f04011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.942+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 shutdown_connections 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.942+0000 7f7f10032700 1 --2- 172.21.15.23:0/2030885525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f08106350 0x7f7f0819dae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.942+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 >> 172.21.15.23:0/2030885525 conn(0x7f7f08101fb0 msgr2=0x7f7f081948b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.942+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 shutdown_connections 2024-03-16T20:26:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:06.942+0000 7f7f10032700 1 -- 172.21.15.23:0/2030885525 wait complete. 2024-03-16T20:26:07.722 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-16T20:26:07.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:07 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:07.719+0000 7f7dc7d32200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T20:26:08.294 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:08 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:08.009+0000 7f7dc7d32200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:08.295 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:08 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:08.292+0000 7f7dc7d32200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:08.587 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:08 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:08.531+0000 7f7dc7d32200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T20:26:09.587 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:09 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:09.367+0000 7f7dc7d32200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T20:26:10.090 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:09 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:09.948+0000 7f7dc7d32200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T20:26:10.090 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:10.049+0000 7f7dc7d32200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T20:26:10.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:10 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3584598787' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T20:26:10.523 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:10.293+0000 7f7dc7d32200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T20:26:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:10.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T20:26:10.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T20:26:10.825 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T20:26:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.827 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T20:25:46.503749+0000", 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe83c5700 1 Processor -- start 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe83c5700 1 -- start start 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe83c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe83c5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe0075840 con 0x7fefe01063b0 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe6161700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe6161700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57750/0 (socket says 172.21.15.23:57750) 2024-03-16T20:26:10.828 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.134+0000 7fefe6161700 1 -- 172.21.15.23:0/2819645273 learned_addr learned my addr 172.21.15.23:0/2819645273 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.135+0000 7fefe6161700 1 -- 172.21.15.23:0/2819645273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe0075980 con 0x7fefe01063b0 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.135+0000 7fefe6161700 1 --2- 172.21.15.23:0/2819645273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0098a0 tx=0x7fefdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02a68859fc24f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.135+0000 7fefe515f700 1 -- 172.21.15.23:0/2819645273 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc004160 con 0x7fefe01063b0 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.135+0000 7fefe515f700 1 -- 172.21.15.23:0/2819645273 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefdc0042c0 con 0x7fefe01063b0 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.135+0000 7fefe515f700 1 -- 172.21.15.23:0/2819645273 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc011ac0 con 0x7fefe01063b0 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- 172.21.15.23:0/2819645273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 msgr2=0x7fefe01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 --2- 172.21.15.23:0/2819645273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0098a0 tx=0x7fefdc009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- 172.21.15.23:0/2819645273 shutdown_connections 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 --2- 172.21.15.23:0/2819645273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe01067d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- 172.21.15.23:0/2819645273 >> 172.21.15.23:0/2819645273 conn(0x7fefe0101fb0 msgr2=0x7fefe01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- 172.21.15.23:0/2819645273 shutdown_connections 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- 172.21.15.23:0/2819645273 wait complete. 2024-03-16T20:26:10.829 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 Processor -- start 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.136+0000 7fefe83c5700 1 -- start start 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe83c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe83c5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe019e040 con 0x7fefe01063b0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe6161700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe6161700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57762/0 (socket says 172.21.15.23:57762) 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe6161700 1 -- 172.21.15.23:0/3584598787 learned_addr learned my addr 172.21.15.23:0/3584598787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe6161700 1 -- 172.21.15.23:0/3584598787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc009550 con 0x7fefe01063b0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe6161700 1 --2- 172.21.15.23:0/3584598787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011a70 tx=0x7fefdc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc0039c0 con 0x7fefe01063b0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefdc003b20 con 0x7fefe01063b0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe019e240 con 0x7fefe01063b0 2024-03-16T20:26:10.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.137+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc024610 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.138+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe019e6e0 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.138+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fefdc024c00 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.138+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fefdc00bab0 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.139+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe01977d0 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.141+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fefdc01aa10 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.235+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fefe0107c80 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.235+0000 7fefd77fe700 1 -- 172.21.15.23:0/3584598787 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fefdc01abb0 con 0x7fefe01063b0 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 msgr2=0x7fefe019db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 --2- 172.21.15.23:0/3584598787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011a70 tx=0x7fefdc011b50 comp rx=0 tx=0).stop 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 shutdown_connections 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 --2- 172.21.15.23:0/3584598787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefe01063b0 0x7fefe019db00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 >> 172.21.15.23:0/3584598787 conn(0x7fefe0101fb0 msgr2=0x7fefe0194910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 shutdown_connections 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:10.236+0000 7fefe83c5700 1 -- 172.21.15.23:0/3584598787 wait complete. 2024-03-16T20:26:10.831 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: Activating manager daemon x 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: mgrmap e2: x(active, starting, since 0.00333909s) 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:26:11.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: Manager daemon x is now available 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' 2024-03-16T20:26:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:11 smithi023 ceph-mon[31879]: from='mgr.14100 172.21.15.23:0/3855743264' entity='mgr.x' 2024-03-16T20:26:12.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:12 smithi023 ceph-mon[31879]: mgrmap e3: x(active, since 1.00742s) 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T20:26:14.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 19, 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T20:26:14.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:14.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T20:26:14.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T20:26:14.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T20:26:14.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T20:26:14.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T20:25:46.503749+0000", 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.267+0000 7f1a4bb68700 1 Processor -- start 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.267+0000 7f1a4bb68700 1 -- start start 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.267+0000 7f1a4bb68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.267+0000 7f1a4bb68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44107420 con 0x7f1a44106a30 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a49904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a49904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59744/0 (socket says 172.21.15.23:59744) 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a49904700 1 -- 172.21.15.23:0/4231344671 learned_addr learned my addr 172.21.15.23:0/4231344671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a49904700 1 -- 172.21.15.23:0/4231344671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a44107c30 con 0x7f1a44106a30 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a49904700 1 --2- 172.21.15.23:0/4231344671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009e80 tx=0x7f1a3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f008f999fa55fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a48902700 1 -- 172.21.15.23:0/4231344671 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a3400e070 con 0x7f1a44106a30 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a48902700 1 -- 172.21.15.23:0/4231344671 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a3400b8c0 con 0x7f1a44106a30 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.268+0000 7f1a48902700 1 -- 172.21.15.23:0/4231344671 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a34004550 con 0x7f1a44106a30 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 -- 172.21.15.23:0/4231344671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 msgr2=0x7f1a44106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/4231344671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009e80 tx=0x7f1a3400b330 comp rx=0 tx=0).stop 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 -- 172.21.15.23:0/4231344671 shutdown_connections 2024-03-16T20:26:14.098 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/4231344671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a44106e50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 -- 172.21.15.23:0/4231344671 >> 172.21.15.23:0/4231344671 conn(0x7f1a44102010 msgr2=0x7f1a44104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 -- 172.21.15.23:0/4231344671 shutdown_connections 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.269+0000 7f1a4bb68700 1 -- 172.21.15.23:0/4231344671 wait complete. 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a4bb68700 1 Processor -- start 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a4bb68700 1 -- start start 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a4bb68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a4bb68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4419be20 con 0x7f1a44106a30 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a49904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a49904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59758/0 (socket says 172.21.15.23:59758) 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a49904700 1 -- 172.21.15.23:0/1667102537 learned_addr learned my addr 172.21.15.23:0/1667102537 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a49904700 1 -- 172.21.15.23:0/1667102537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a34009b30 con 0x7f1a44106a30 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a49904700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009fd0 tx=0x7f1a34003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.270+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a3400e040 con 0x7f1a44106a30 2024-03-16T20:26:14.099 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.271+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a340128f0 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.271+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4419c020 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.271+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a3401c770 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.271+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4419c4a0 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.271+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 3) v1 ==== 45652+0+0 (secure 0 0 0) 0x7f1a3401c9e0 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.272+0000 7f1a42ffd700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 0x7f1a2c03b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.272+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a44195ad0 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.272+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1a3404cb00 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.272+0000 7f1a49103700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 0x7f1a2c03b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.272+0000 7f1a49103700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 0x7f1a2c03b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a38006fd0 tx=0x7f1a38006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.277+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1a34011390 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.317+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f1a3401cc90 con 0x7f1a44106a30 2024-03-16T20:26:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.564+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1a4402dab0 con 0x7f1a44106a30 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.564+0000 7f1a42ffd700 1 -- 172.21.15.23:0/1667102537 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f1a34011390 con 0x7f1a44106a30 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.566+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 msgr2=0x7f1a2c03b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.566+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 0x7f1a2c03b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a38006fd0 tx=0x7f1a38006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.566+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 msgr2=0x7f1a4419b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.566+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009fd0 tx=0x7f1a34003d00 comp rx=0 tx=0).stop 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 shutdown_connections 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f1a2c038b90 0x7f1a2c03b050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 --2- 172.21.15.23:0/1667102537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a44106a30 0x7f1a4419b8e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 >> 172.21.15.23:0/1667102537 conn(0x7f1a44102010 msgr2=0x7f1a44102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 shutdown_connections 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:13.567+0000 7f1a4bb68700 1 -- 172.21.15.23:0/1667102537 wait complete. 2024-03-16T20:26:14.101 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-16T20:26:14.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:14 smithi023 ceph-mon[31879]: mgrmap e4: x(active, since 3s) 2024-03-16T20:26:14.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:14 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1667102537' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:15.305 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a6d26700 1 Processor -- start 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a6d26700 1 -- start start 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a6d26700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:15.306 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a6d26700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0106770 con 0x7ff4a0105d80 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a4ac2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a4ac2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59760/0 (socket says 172.21.15.23:59760) 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a4ac2700 1 -- 172.21.15.23:0/865504195 learned_addr learned my addr 172.21.15.23:0/865504195 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.522+0000 7ff4a4ac2700 1 -- 172.21.15.23:0/865504195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a01068b0 con 0x7ff4a0105d80 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a4ac2700 1 --2- 172.21.15.23:0/865504195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff4940098a0 tx=0x7ff494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cebf6c45b8934e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff49f7fe700 1 -- 172.21.15.23:0/865504195 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff494004160 con 0x7ff4a0105d80 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff49f7fe700 1 -- 172.21.15.23:0/865504195 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4940042c0 con 0x7ff4a0105d80 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff49f7fe700 1 -- 172.21.15.23:0/865504195 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff494011ac0 con 0x7ff4a0105d80 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 -- 172.21.15.23:0/865504195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 msgr2=0x7ff4a01061a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/865504195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff4940098a0 tx=0x7ff494009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 -- 172.21.15.23:0/865504195 shutdown_connections 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/865504195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a01061a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:15.307 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 -- 172.21.15.23:0/865504195 >> 172.21.15.23:0/865504195 conn(0x7ff4a01017a0 msgr2=0x7ff4a0103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.523+0000 7ff4a6d26700 1 -- 172.21.15.23:0/865504195 shutdown_connections 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a6d26700 1 -- 172.21.15.23:0/865504195 wait complete. 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a6d26700 1 Processor -- start 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a6d26700 1 -- start start 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a6d26700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a6d26700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a019be60 con 0x7ff4a0105d80 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a4ac2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a4ac2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59772/0 (socket says 172.21.15.23:59772) 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.524+0000 7ff4a4ac2700 1 -- 172.21.15.23:0/1171781660 learned_addr learned my addr 172.21.15.23:0/1171781660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff4a4ac2700 1 -- 172.21.15.23:0/1171781660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494009550 con 0x7ff4a0105d80 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff4a4ac2700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff494009fb0 tx=0x7ff494011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4940038b0 con 0x7ff4a0105d80 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff494003a10 con 0x7ff4a0105d80 2024-03-16T20:26:15.308 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a019c060 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4940195e0 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.525+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a019c500 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.526+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7ff49401aa10 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.526+0000 7ff49dffb700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 0x7ff49003b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.526+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff49404cd90 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.526+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a0109ae0 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.526+0000 7ff49ffff700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 0x7ff49003b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.527+0000 7ff49ffff700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 0x7ff49003b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff48c006fd0 tx=0x7ff48c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.531+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff494019850 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.736+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff4a019ede0 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.737+0000 7ff49dffb700 1 -- 172.21.15.23:0/1171781660 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7ff494023c20 con 0x7ff4a0105d80 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 msgr2=0x7ff49003b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:15.309 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 0x7ff49003b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff48c006fd0 tx=0x7ff48c006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 msgr2=0x7ff4a019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff494009fb0 tx=0x7ff494011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 shutdown_connections 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7ff490038c60 0x7ff49003b120 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 --2- 172.21.15.23:0/1171781660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a0105d80 0x7ff4a019b920 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.739+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 >> 172.21.15.23:0/1171781660 conn(0x7ff4a01017a0 msgr2=0x7ff4a0102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.740+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 shutdown_connections 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:14.740+0000 7ff4a6d26700 1 -- 172.21.15.23:0/1171781660 wait complete. 2024-03-16T20:26:15.310 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-16T20:26:15.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1171781660' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T20:26:16.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:16 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2587281571' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T20:26:16.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:16 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: ignoring --setuser ceph since I am not root 2024-03-16T20:26:16.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:16 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: ignoring --setgroup ceph since I am not root 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.681+0000 7f299d5f1700 1 Processor -- start 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.681+0000 7f299d5f1700 1 -- start start 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.681+0000 7f299d5f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.681+0000 7f299d5f1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29981084c0 con 0x7f2998107b60 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.681+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59780/0 (socket says 172.21.15.23:59780) 2024-03-16T20:26:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2996ffd700 1 -- 172.21.15.23:0/1750565922 learned_addr learned my addr 172.21.15.23:0/1750565922 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2996ffd700 1 -- 172.21.15.23:0/1750565922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998108600 con 0x7f2998107b60 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2996ffd700 1 --2- 172.21.15.23:0/1750565922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2990009e80 tx=0x7f299000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb0d2ac86edfa1a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2995ffb700 1 -- 172.21.15.23:0/1750565922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f299000e070 con 0x7f2998107b60 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2995ffb700 1 -- 172.21.15.23:0/1750565922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f299000b8c0 con 0x7f2998107b60 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.682+0000 7f2995ffb700 1 -- 172.21.15.23:0/1750565922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2990004550 con 0x7f2998107b60 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 -- 172.21.15.23:0/1750565922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 msgr2=0x7f2998107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 --2- 172.21.15.23:0/1750565922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2990009e80 tx=0x7f299000b330 comp rx=0 tx=0).stop 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 -- 172.21.15.23:0/1750565922 shutdown_connections 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 --2- 172.21.15.23:0/1750565922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f2998107f80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 -- 172.21.15.23:0/1750565922 >> 172.21.15.23:0/1750565922 conn(0x7f29981031c0 msgr2=0x7f29981055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 -- 172.21.15.23:0/1750565922 shutdown_connections 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 -- 172.21.15.23:0/1750565922 wait complete. 2024-03-16T20:26:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.683+0000 7f299d5f1700 1 Processor -- start 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f299d5f1700 1 -- start start 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f299d5f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f299d5f1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299807e030 con 0x7f2998107b60 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f2996ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59794/0 (socket says 172.21.15.23:59794) 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f2996ffd700 1 -- 172.21.15.23:0/2587281571 learned_addr learned my addr 172.21.15.23:0/2587281571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f2996ffd700 1 -- 172.21.15.23:0/2587281571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990009b30 con 0x7f2998107b60 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f2996ffd700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f2990003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f299000e040 con 0x7f2998107b60 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29900128f0 con 0x7f2998107b60 2024-03-16T20:26:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299807c220 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f299001c890 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.684+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299807c6a0 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.685+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f299001cb00 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.685+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998063610 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.685+0000 7f298ffff700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 0x7f298403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.686+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f299004ccf0 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.686+0000 7f29967fc700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 0x7f298403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.686+0000 7f29967fc700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 0x7f298403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f2980006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.690+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2990011940 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:15.951+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2998109990 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.328+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2990017070 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.330+0000 7f298ffff700 1 -- 172.21.15.23:0/2587281571 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f2990024d00 con 0x7f2998107b60 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 msgr2=0x7f298403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 0x7f298403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2980006fd0 tx=0x7f2980006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 msgr2=0x7f299807daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f2990003d70 comp rx=0 tx=0).stop 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 shutdown_connections 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2984038c60 0x7f298403b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 --2- 172.21.15.23:0/2587281571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2998107b60 0x7f299807daf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 >> 172.21.15.23:0/2587281571 conn(0x7f29981031c0 msgr2=0x7f2998103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 shutdown_connections 2024-03-16T20:26:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:16.331+0000 7f299d5f1700 1 -- 172.21.15.23:0/2587281571 wait complete. 2024-03-16T20:26:17.175 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:16 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:16.902+0000 7fe1e1036200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T20:26:17.176 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:17.007+0000 7fe1e1036200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T20:26:17.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2587281571' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T20:26:17.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-mon[31879]: mgrmap e5: x(active, since 6s) 2024-03-16T20:26:17.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:17.677+0000 7fe1e1036200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T20:26:17.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:17.786+0000 7fe1e1036200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T20:26:18.231 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:17 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:17.930+0000 7fe1e1036200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T20:26:18.232 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.029+0000 7fe1e1036200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T20:26:18.232 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.114+0000 7fe1e1036200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T20:26:18.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:18.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T20:26:18.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.374+0000 7f2e38a15700 1 Processor -- start 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e38a15700 1 -- start start 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e38a15700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e38a15700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341084d0 con 0x7f2e34107ae0 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59806/0 (socket says 172.21.15.23:59806) 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e3259c700 1 -- 172.21.15.23:0/3246475933 learned_addr learned my addr 172.21.15.23:0/3246475933 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.375+0000 7f2e3259c700 1 -- 172.21.15.23:0/3246475933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e34108ce0 con 0x7f2e34107ae0 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.376+0000 7f2e3259c700 1 --2- 172.21.15.23:0/3246475933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0098a0 tx=0x7f2e1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f50d5a2aac46bb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:18.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.376+0000 7f2e3159a700 1 -- 172.21.15.23:0/3246475933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c004160 con 0x7f2e34107ae0 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.376+0000 7f2e3159a700 1 -- 172.21.15.23:0/3246475933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e1c0042c0 con 0x7f2e34107ae0 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.376+0000 7f2e3159a700 1 -- 172.21.15.23:0/3246475933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c011ac0 con 0x7f2e34107ae0 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 -- 172.21.15.23:0/3246475933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 msgr2=0x7f2e34107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 --2- 172.21.15.23:0/3246475933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0098a0 tx=0x7f2e1c009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 -- 172.21.15.23:0/3246475933 shutdown_connections 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 --2- 172.21.15.23:0/3246475933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e34107f00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 -- 172.21.15.23:0/3246475933 >> 172.21.15.23:0/3246475933 conn(0x7f2e34103040 msgr2=0x7f2e341054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 -- 172.21.15.23:0/3246475933 shutdown_connections 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.377+0000 7f2e38a15700 1 -- 172.21.15.23:0/3246475933 wait complete. 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e38a15700 1 Processor -- start 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e38a15700 1 -- start start 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e38a15700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e38a15700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3419be90 con 0x7f2e34107ae0 2024-03-16T20:26:18.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59814/0 (socket says 172.21.15.23:59814) 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e3259c700 1 -- 172.21.15.23:0/1759334941 learned_addr learned my addr 172.21.15.23:0/1759334941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e3259c700 1 -- 172.21.15.23:0/1759334941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c009550 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e3259c700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009fb0 tx=0x7f2e1c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.378+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c0038b0 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.379+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3419c090 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.379+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e1c003a10 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.379+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1c0195e0 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.379+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3419c530 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.379+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f2e1c019850 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.380+0000 7f2e2b7fe700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 0x7f2e2003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.380+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2e1c04ddf0 con 0x7f2e34107ae0 2024-03-16T20:26:18.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.380+0000 7f2e31d9b700 1 -- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 msgr2=0x7f2e2003b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.380+0000 7f2e31d9b700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 0x7f2e2003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.380+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3410bf10 con 0x7f2e34107ae0 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.389+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2e1c019b50 con 0x7f2e34107ae0 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.581+0000 7f2e31d9b700 1 -- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 msgr2=0x7f2e2003b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.581+0000 7f2e31d9b700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 0x7f2e2003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.653+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2e3419f080 con 0x7f2e34107ae0 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.653+0000 7f2e2b7fe700 1 -- 172.21.15.23:0/1759334941 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2e1c018be0 con 0x7f2e34107ae0 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 msgr2=0x7f2e2003b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 0x7f2e2003b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 msgr2=0x7f2e3419b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009fb0 tx=0x7f2e1c011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 shutdown_connections 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7f2e20038cb0 0x7f2e2003b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:18.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 --2- 172.21.15.23:0/1759334941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e34107ae0 0x7f2e3419b950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:18.244 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 >> 172.21.15.23:0/1759334941 conn(0x7f2e34103040 msgr2=0x7f2e34103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:18.244 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 shutdown_connections 2024-03-16T20:26:18.244 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:17.656+0000 7f2e38a15700 1 -- 172.21.15.23:0/1759334941 wait complete. 2024-03-16T20:26:18.244 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-16T20:26:18.244 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-16T20:26:18.505 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1759334941' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T20:26:18.505 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.387+0000 7fe1e1036200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T20:26:18.839 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.502+0000 7fe1e1036200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:18.839 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.593+0000 7fe1e1036200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T20:26:18.839 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.693+0000 7fe1e1036200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T20:26:19.249 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:18 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:18.871+0000 7fe1e1036200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T20:26:20.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:19 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:19.854+0000 7fe1e1036200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T20:26:20.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:19 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:19.988+0000 7fe1e1036200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T20:26:20.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:20 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:20.328+0000 7fe1e1036200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T20:26:20.587 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:20 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:20.419+0000 7fe1e1036200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T20:26:21.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:20 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:20.618+0000 7fe1e1036200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:21.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:20 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:20.807+0000 7fe1e1036200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T20:26:21.362 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:21.098+0000 7fe1e1036200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:21.362 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:21.359+0000 7fe1e1036200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:21.650 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:21.599+0000 7fe1e1036200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T20:26:22.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:22 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:22.419+0000 7fe1e1036200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T20:26:23.323 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:22.999+0000 7fe1e1036200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T20:26:23.323 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:23.100+0000 7fe1e1036200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T20:26:23.323 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:23.320+0000 7fe1e1036200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: Active manager daemon x restarted 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: Activating manager daemon x 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: mgrmap e6: x(active, starting, since 0.00815656s) 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T20:26:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: Manager daemon x is now available 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: Found migration_current of "None". Setting to last migration. 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:23 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: mgrmap e7: x(active, since 1.01025s) 2024-03-16T20:26:24.796 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:24 smithi023 ceph-mon[31879]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T20:26:25.081 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6ff012700 1 Processor -- start 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6ff012700 1 -- start start 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6ff012700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6ff012700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f80758a0 con 0x7fd6f8108560 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6fcdae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6fcdae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59830/0 (socket says 172.21.15.23:59830) 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.640+0000 7fd6fcdae700 1 -- 172.21.15.23:0/1273368571 learned_addr learned my addr 172.21.15.23:0/1273368571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6fcdae700 1 -- 172.21.15.23:0/1273368571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f810ae90 con 0x7fd6f8108560 2024-03-16T20:26:25.082 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6fcdae700 1 --2- 172.21.15.23:0/1273368571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec0098a0 tx=0x7fd6ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c753746c52b6482a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6f77fe700 1 -- 172.21.15.23:0/1273368571 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec004160 con 0x7fd6f8108560 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6f77fe700 1 -- 172.21.15.23:0/1273368571 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6ec0042c0 con 0x7fd6f8108560 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6f77fe700 1 -- 172.21.15.23:0/1273368571 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec011ac0 con 0x7fd6f8108560 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6ff012700 1 -- 172.21.15.23:0/1273368571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 msgr2=0x7fd6f810a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.641+0000 7fd6ff012700 1 --2- 172.21.15.23:0/1273368571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec0098a0 tx=0x7fd6ec009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- 172.21.15.23:0/1273368571 shutdown_connections 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 --2- 172.21.15.23:0/1273368571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f810a950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- 172.21.15.23:0/1273368571 >> 172.21.15.23:0/1273368571 conn(0x7fd6f8101fb0 msgr2=0x7fd6f81043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- 172.21.15.23:0/1273368571 shutdown_connections 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- 172.21.15.23:0/1273368571 wait complete. 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 Processor -- start 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- start start 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:25.083 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6ff012700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8199c40 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6fcdae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6fcdae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59834/0 (socket says 172.21.15.23:59834) 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.642+0000 7fd6fcdae700 1 -- 172.21.15.23:0/2355076773 learned_addr learned my addr 172.21.15.23:0/2355076773 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6fcdae700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec009550 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6fcdae700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009fb0 tx=0x7fd6ec011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec0038b0 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8199e40 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6ec003a10 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec0195e0 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.643+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f819a2e0 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fd6ec019850 con 0x7fd6f8108560 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6f5ffb700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:25.084 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6f8106b90 con 0x7fd6e803d0e0 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6ec04ce00 con 0x7fd6f8108560 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6f7fff700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.644+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.845+0000 7fd6f7fff700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:18.845+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:19.246+0000 7fd6f7fff700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:19.246+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:20.047+0000 7fd6f7fff700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:20.047+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:21.648+0000 7fd6f7fff700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3890527981 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:21.648+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:23.340+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7fd6ec02d960 con 0x7fd6f8108560 2024-03-16T20:26:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:23.340+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 msgr2=0x7fd6e803f5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:23.340+0000 7fd6f5ffb700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.341+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fd6ec00b930 con 0x7fd6f8108560 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.341+0000 7fd6f5ffb700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 0x7fd6e8042c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.341+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6f8106b90 con 0x7fd6e80408a0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.342+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 0x7fd6e8042c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.343+0000 7fd6f7fff700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 0x7fd6e8042c90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003a10 tx=0x7fd6e40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.343+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fd6f8106b90 con 0x7fd6e80408a0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd6f8106d00 con 0x7fd6e80408a0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6f5ffb700 1 -- 172.21.15.23:0/2355076773 <== mgr.14120 v2:172.21.15.23:6800/2899821830 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd6f8106d00 con 0x7fd6e80408a0 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 msgr2=0x7fd6e8042c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 0x7fd6e8042c90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003a10 tx=0x7fd6e40092b0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 msgr2=0x7fd6f8199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:25.086 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009fb0 tx=0x7fd6ec011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 shutdown_connections 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fd6e80408a0 0x7fd6e8042c90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f8108560 0x7fd6f8199700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 --2- 172.21.15.23:0/2355076773 >> [v2:172.21.15.23:6800/3890527981,v1:172.21.15.23:6801/3890527981] conn(0x7fd6e803d0e0 0x7fd6e803f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 >> 172.21.15.23:0/2355076773 conn(0x7fd6f8101fb0 msgr2=0x7fd6f8190160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 shutdown_connections 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:24.350+0000 7fd6ff012700 1 -- 172.21.15.23:0/2355076773 wait complete. 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-16T20:26:25.087 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-16T20:26:26.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:25 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:26.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:25 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:26.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:25 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:26.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:25 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:26.259 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb41295700 1 Processor -- start 2024-03-16T20:26:26.259 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb41295700 1 -- start start 2024-03-16T20:26:26.259 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb41295700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb41295700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c108520 con 0x7feb3c107b30 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60352/0 (socket says 172.21.15.23:60352) 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb3affd700 1 -- 172.21.15.23:0/1082385864 learned_addr learned my addr 172.21.15.23:0/1082385864 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb3affd700 1 -- 172.21.15.23:0/1082385864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c108ce0 con 0x7feb3c107b30 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb3affd700 1 --2- 172.21.15.23:0/1082385864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098a0 tx=0x7feb2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958954867ff5a74f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.472+0000 7feb39ffb700 1 -- 172.21.15.23:0/1082385864 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb2c004160 con 0x7feb3c107b30 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb39ffb700 1 -- 172.21.15.23:0/1082385864 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb2c0042c0 con 0x7feb3c107b30 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb39ffb700 1 -- 172.21.15.23:0/1082385864 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb2c011ac0 con 0x7feb3c107b30 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 -- 172.21.15.23:0/1082385864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 msgr2=0x7feb3c107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 --2- 172.21.15.23:0/1082385864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098a0 tx=0x7feb2c009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 -- 172.21.15.23:0/1082385864 shutdown_connections 2024-03-16T20:26:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 --2- 172.21.15.23:0/1082385864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c107f50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 -- 172.21.15.23:0/1082385864 >> 172.21.15.23:0/1082385864 conn(0x7feb3c103110 msgr2=0x7feb3c105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 -- 172.21.15.23:0/1082385864 shutdown_connections 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.473+0000 7feb41295700 1 -- 172.21.15.23:0/1082385864 wait complete. 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb41295700 1 Processor -- start 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb41295700 1 -- start start 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb41295700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb41295700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c07e030 con 0x7feb3c107b30 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60358/0 (socket says 172.21.15.23:60358) 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.474+0000 7feb3affd700 1 -- 172.21.15.23:0/2402008964 learned_addr learned my addr 172.21.15.23:0/2402008964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb3affd700 1 -- 172.21.15.23:0/2402008964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c009550 con 0x7feb3c107b30 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb3affd700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb2c011780 tx=0x7feb2c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb2c003af0 con 0x7feb3c107b30 2024-03-16T20:26:26.261 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c07c220 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb2c003c50 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb2c0197e0 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.475+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c07c6c0 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.476+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7feb2c019a50 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.476+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c10bf10 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.476+0000 7feb23fff700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 0x7feb2403b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.476+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb2c04cf40 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.476+0000 7feb3a7fc700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 0x7feb2403b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.477+0000 7feb3a7fc700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 0x7feb2403b0a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb30006fd0 tx=0x7feb30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.481+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7feb2c018a40 con 0x7feb3c107b30 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.704+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7feb3c07cdb0 con 0x7feb24038be0 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.708+0000 7feb23fff700 1 -- 172.21.15.23:0/2402008964 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feb3c07cdb0 con 0x7feb24038be0 2024-03-16T20:26:26.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 msgr2=0x7feb2403b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 0x7feb2403b0a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb30006fd0 tx=0x7feb30006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 msgr2=0x7feb3c07daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb2c011780 tx=0x7feb2c011b40 comp rx=0 tx=0).stop 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 shutdown_connections 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7feb24038be0 0x7feb2403b0a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 --2- 172.21.15.23:0/2402008964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb3c107b30 0x7feb3c07daf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 >> 172.21.15.23:0/2402008964 conn(0x7feb3c103110 msgr2=0x7feb3c103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 shutdown_connections 2024-03-16T20:26:26.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:25.711+0000 7feb41295700 1 -- 172.21.15.23:0/2402008964 wait complete. 2024-03-16T20:26:27.191 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:25] ENGINE Bus STARTING 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:25] ENGINE Serving on http://172.21.15.23:8765 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: mgrmap e8: x(active, since 2s) 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:26] ENGINE Serving on https://172.21.15.23:7150 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:26] ENGINE Bus STARTED 2024-03-16T20:26:27.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:26 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:27.450 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T20:26:27.450 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92d0123700 1 Processor -- start 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92d0123700 1 -- start start 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92d0123700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92d0123700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c81078c0 con 0x7f92c8106ed0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92cdebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92cdebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60368/0 (socket says 172.21.15.23:60368) 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.679+0000 7f92cdebf700 1 -- 172.21.15.23:0/1378019291 learned_addr learned my addr 172.21.15.23:0/1378019291 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.680+0000 7f92cdebf700 1 -- 172.21.15.23:0/1378019291 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c8107bf0 con 0x7f92c8106ed0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.680+0000 7f92cdebf700 1 --2- 172.21.15.23:0/1378019291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92c400bb40 tx=0x7f92c400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe43739a83ab511b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.680+0000 7f92ccebd700 1 -- 172.21.15.23:0/1378019291 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c400dbe0 con 0x7f92c8106ed0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.680+0000 7f92ccebd700 1 -- 172.21.15.23:0/1378019291 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92c40044f0 con 0x7f92c8106ed0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.680+0000 7f92ccebd700 1 -- 172.21.15.23:0/1378019291 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c4010470 con 0x7f92c8106ed0 2024-03-16T20:26:27.451 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- 172.21.15.23:0/1378019291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 msgr2=0x7f92c81072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 --2- 172.21.15.23:0/1378019291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92c400bb40 tx=0x7f92c400be50 comp rx=0 tx=0).stop 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- 172.21.15.23:0/1378019291 shutdown_connections 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 --2- 172.21.15.23:0/1378019291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c81072f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- 172.21.15.23:0/1378019291 >> 172.21.15.23:0/1378019291 conn(0x7f92c8101730 msgr2=0x7f92c8103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- 172.21.15.23:0/1378019291 shutdown_connections 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- 172.21.15.23:0/1378019291 wait complete. 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 Processor -- start 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- start start 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.681+0000 7f92d0123700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c819dfb0 con 0x7f92c8106ed0 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92cdebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92cdebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60372/0 (socket says 172.21.15.23:60372) 2024-03-16T20:26:27.452 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92cdebf700 1 -- 172.21.15.23:0/2266384895 learned_addr learned my addr 172.21.15.23:0/2266384895 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92cdebf700 1 -- 172.21.15.23:0/2266384895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c400b7f0 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92cdebf700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92c400d650 tx=0x7f92c40040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c40106f0 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92c4010d10 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c819e1b0 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92c4019d40 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.682+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c819e5f0 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.683+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f92c402a450 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.683+0000 7f92beffd700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 0x7f92b403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.683+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8105a30 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.683+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f92c404d260 con 0x7f92c8106ed0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.683+0000 7f92cd6be700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 0x7f92b403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:27.453 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.684+0000 7f92cd6be700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 0x7f92b403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92b8006fd0 tx=0x7f92b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.688+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f92c400fbd0 con 0x7f92c8106ed0 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.897+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f92c8062420 con 0x7f92b4038c60 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.897+0000 7f92beffd700 1 -- 172.21.15.23:0/2266384895 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f92c8062420 con 0x7f92b4038c60 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 msgr2=0x7f92b403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 0x7f92b403b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92b8006fd0 tx=0x7f92b8006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 msgr2=0x7f92c819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92c400d650 tx=0x7f92c40040b0 comp rx=0 tx=0).stop 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 shutdown_connections 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f92b4038c60 0x7f92b403b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 --2- 172.21.15.23:0/2266384895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8106ed0 0x7f92c819da70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 >> 172.21.15.23:0/2266384895 conn(0x7f92c8101730 msgr2=0x7f92c818fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 shutdown_connections 2024-03-16T20:26:27.454 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:26.899+0000 7f92d0123700 1 -- 172.21.15.23:0/2266384895 wait complete. 2024-03-16T20:26:27.455 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-16T20:26:28.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:27 smithi023 ceph-mon[31879]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:28.744 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: Generating public/private rsa key pair. 2024-03-16T20:26:28.744 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: Your identification has been saved in /tmp/tmphphtt2t3/key. 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: Your public key has been saved in /tmp/tmphphtt2t3/key.pub. 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: The key fingerprint is: 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: SHA256:c3Ce4VkNGN4hoZlq2HL044FBJlYSmkmQVBQ/Jtyw6OE ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: The key's randomart image is: 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: +---[RSA 3072]----+ 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: |o++=*o+ ++o | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: |..o=== =.o + | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | o++ =o = + o . | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: |o . o+.= = = | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | E o * S * | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | + . = | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | . | 2024-03-16T20:26:28.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | | 2024-03-16T20:26:28.746 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: | | 2024-03-16T20:26:28.746 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: +----[SHA256]-----+ 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f60260d4700 1 Processor -- start 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f60260d4700 1 -- start start 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f60260d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f60260d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6020107430 con 0x7f6020106a40 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f601f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:29.007 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f601f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60384/0 (socket says 172.21.15.23:60384) 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f601f7fe700 1 -- 172.21.15.23:0/3905638308 learned_addr learned my addr 172.21.15.23:0/3905638308 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f601f7fe700 1 -- 172.21.15.23:0/3905638308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020107c40 con 0x7f6020106a40 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.846+0000 7f601f7fe700 1 --2- 172.21.15.23:0/3905638308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60100098a0 tx=0x7f6010009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5821228e6e4449b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f601e7fc700 1 -- 172.21.15.23:0/3905638308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6010004160 con 0x7f6020106a40 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f601e7fc700 1 -- 172.21.15.23:0/3905638308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60100042c0 con 0x7f6020106a40 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f601e7fc700 1 -- 172.21.15.23:0/3905638308 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6010011ac0 con 0x7f6020106a40 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 -- 172.21.15.23:0/3905638308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 msgr2=0x7f6020106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 --2- 172.21.15.23:0/3905638308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60100098a0 tx=0x7f6010009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 -- 172.21.15.23:0/3905638308 shutdown_connections 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 --2- 172.21.15.23:0/3905638308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f6020106e60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 -- 172.21.15.23:0/3905638308 >> 172.21.15.23:0/3905638308 conn(0x7f6020102020 msgr2=0x7f6020104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 -- 172.21.15.23:0/3905638308 shutdown_connections 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.847+0000 7f60260d4700 1 -- 172.21.15.23:0/3905638308 wait complete. 2024-03-16T20:26:29.008 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f60260d4700 1 Processor -- start 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f60260d4700 1 -- start start 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f60260d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f60260d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019be90 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f601f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f601f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60396/0 (socket says 172.21.15.23:60396) 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.848+0000 7f601f7fe700 1 -- 172.21.15.23:0/3874848989 learned_addr learned my addr 172.21.15.23:0/3874848989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f601f7fe700 1 -- 172.21.15.23:0/3874848989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010009550 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f601f7fe700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6010009fb0 tx=0x7f6010011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60100038b0 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6010003a10 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602019c090 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60100195e0 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.849+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602019c530 con 0x7f6020106a40 2024-03-16T20:26:29.009 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.850+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f60100198c0 con 0x7f6020106a40 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.850+0000 7f601cff9700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 0x7f600803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.850+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f601004ce40 con 0x7f6020106a40 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.850+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6020195ae0 con 0x7f6020106a40 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.850+0000 7f601effd700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 0x7f600803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.851+0000 7f601effd700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 0x7f600803b120 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6014006fd0 tx=0x7f6014006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:27.855+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6010019bc0 con 0x7f6020106a40 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.063+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f602002dad0 con 0x7f6008038c60 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.453+0000 7f601cff9700 1 -- 172.21.15.23:0/3874848989 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f602002dad0 con 0x7f6008038c60 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 msgr2=0x7f600803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 0x7f600803b120 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6014006fd0 tx=0x7f6014006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 msgr2=0x7f602019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6010009fb0 tx=0x7f6010011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:29.010 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 shutdown_connections 2024-03-16T20:26:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f6008038c60 0x7f600803b120 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 --2- 172.21.15.23:0/3874848989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6020106a40 0x7f602019b950 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.455+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 >> 172.21.15.23:0/3874848989 conn(0x7f6020102020 msgr2=0x7f6020102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.456+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 shutdown_connections 2024-03-16T20:26:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:28.456+0000 7f60260d4700 1 -- 172.21.15.23:0/3874848989 wait complete. 2024-03-16T20:26:29.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-mon[31879]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:29.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-mon[31879]: Generating ssh key... 2024-03-16T20:26:29.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:29.327 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:28 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:30.216 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:30.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.424+0000 7f2bb861c700 1 Processor -- start 2024-03-16T20:26:30.222 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.424+0000 7f2bb861c700 1 -- start start 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.424+0000 7f2bb861c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.424+0000 7f2bb861c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb007bdb0 con 0x7f2bb0108670 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb63b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb63b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60412/0 (socket says 172.21.15.23:60412) 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb63b8700 1 -- 172.21.15.23:0/4268004194 learned_addr learned my addr 172.21.15.23:0/4268004194 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb63b8700 1 -- 172.21.15.23:0/4268004194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb010afa0 con 0x7f2bb0108670 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb63b8700 1 --2- 172.21.15.23:0/4268004194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0098a0 tx=0x7f2bac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e835e7e8b9e07 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb53b6700 1 -- 172.21.15.23:0/4268004194 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bac004160 con 0x7f2bb0108670 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb53b6700 1 -- 172.21.15.23:0/4268004194 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bac0042c0 con 0x7f2bb0108670 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.425+0000 7f2bb53b6700 1 -- 172.21.15.23:0/4268004194 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bac011ac0 con 0x7f2bb0108670 2024-03-16T20:26:30.223 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 -- 172.21.15.23:0/4268004194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 msgr2=0x7f2bb010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 --2- 172.21.15.23:0/4268004194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0098a0 tx=0x7f2bac009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 -- 172.21.15.23:0/4268004194 shutdown_connections 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 --2- 172.21.15.23:0/4268004194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb010aa60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 -- 172.21.15.23:0/4268004194 >> 172.21.15.23:0/4268004194 conn(0x7f2bb0101fe0 msgr2=0x7f2bb0104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 -- 172.21.15.23:0/4268004194 shutdown_connections 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 -- 172.21.15.23:0/4268004194 wait complete. 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.426+0000 7f2bb861c700 1 Processor -- start 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb861c700 1 -- start start 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb861c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb861c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb0108430 con 0x7f2bb0108670 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb63b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb63b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60416/0 (socket says 172.21.15.23:60416) 2024-03-16T20:26:30.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb63b8700 1 -- 172.21.15.23:0/1772671648 learned_addr learned my addr 172.21.15.23:0/1772671648 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb63b8700 1 -- 172.21.15.23:0/1772671648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac009550 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb63b8700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2bac011a70 tx=0x7f2bac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bac0039c0 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bac003b20 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bac024610 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb0106620 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.427+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb0106ac0 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.428+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f2bac01a430 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.429+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb010bd50 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.429+0000 7f2ba77fe700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 0x7f2b9c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.429+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2bac04cc50 con 0x7f2bb0108670 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.429+0000 7f2bb5bb7700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 0x7f2b9c03b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:30.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.429+0000 7f2bb5bb7700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 0x7f2b9c03b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006fd0 tx=0x7f2ba0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.434+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2bac010380 con 0x7f2bb0108670 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.639+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bb0107100 con 0x7f2b9c038d00 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.640+0000 7f2ba77fe700 1 -- 172.21.15.23:0/1772671648 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2bb0107100 con 0x7f2b9c038d00 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 msgr2=0x7f2b9c03b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 0x7f2b9c03b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006fd0 tx=0x7f2ba0006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 msgr2=0x7f2bb0107ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2bac011a70 tx=0x7f2bac011b50 comp rx=0 tx=0).stop 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 shutdown_connections 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2b9c038d00 0x7f2b9c03b1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 --2- 172.21.15.23:0/1772671648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0108670 0x7f2bb0107ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 >> 172.21.15.23:0/1772671648 conn(0x7f2bb0101fe0 msgr2=0x7f2bb0102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 shutdown_connections 2024-03-16T20:26:30.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:29.642+0000 7f2bb861c700 1 -- 172.21.15.23:0/1772671648 wait complete. 2024-03-16T20:26:30.227 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T20:26:30.227 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T20:26:30.227 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-16T20:26:30.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:30 smithi023 ceph-mon[31879]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:31.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:31 smithi023 ceph-mon[31879]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:32.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:32 smithi023 ceph-mon[31879]: Deploying cephadm binary to smithi023 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022f981700 1 Processor -- start 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022f981700 1 -- start start 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022f981700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022f981700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022807bdf0 con 0x7f02281085a0 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022d71d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022d71d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36200/0 (socket says 172.21.15.23:36200) 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022d71d700 1 -- 172.21.15.23:0/131764848 learned_addr learned my addr 172.21.15.23:0/131764848 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.626+0000 7f022d71d700 1 -- 172.21.15.23:0/131764848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022810aed0 con 0x7f02281085a0 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022d71d700 1 --2- 172.21.15.23:0/131764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f021c0098a0 tx=0x7f021c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812ed6c94f754127 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:35.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f021bfff700 1 -- 172.21.15.23:0/131764848 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f021c004160 con 0x7f02281085a0 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f021bfff700 1 -- 172.21.15.23:0/131764848 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021c0042c0 con 0x7f02281085a0 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f021bfff700 1 -- 172.21.15.23:0/131764848 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f021c011ac0 con 0x7f02281085a0 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022f981700 1 -- 172.21.15.23:0/131764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 msgr2=0x7f022810a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022f981700 1 --2- 172.21.15.23:0/131764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f021c0098a0 tx=0x7f021c009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022f981700 1 -- 172.21.15.23:0/131764848 shutdown_connections 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022f981700 1 --2- 172.21.15.23:0/131764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f022810a990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.627+0000 7f022f981700 1 -- 172.21.15.23:0/131764848 >> 172.21.15.23:0/131764848 conn(0x7f0228101fd0 msgr2=0x7f02281043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 -- 172.21.15.23:0/131764848 shutdown_connections 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 -- 172.21.15.23:0/131764848 wait complete. 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 Processor -- start 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 -- start start 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:35.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022f981700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228108360 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022d71d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022d71d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36210/0 (socket says 172.21.15.23:36210) 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.628+0000 7f022d71d700 1 -- 172.21.15.23:0/1966684714 learned_addr learned my addr 172.21.15.23:0/1966684714 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f022d71d700 1 -- 172.21.15.23:0/1966684714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c009550 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f022d71d700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f021c009fb0 tx=0x7f021c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f021c0038b0 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0228106550 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021c003a10 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f021c0195e0 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.629+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02281069f0 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f021c019870 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022810bcd0 con 0x7f02281085a0 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f021a7fc700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 0x7f021403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:35.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f021c04cf50 con 0x7f02281085a0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f022cf1c700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 0x7f021403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.630+0000 7f022cf1c700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 0x7f021403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0224006fd0 tx=0x7f0224006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.635+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f021c018530 con 0x7f02281085a0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:30.844+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7f0228107080 con 0x7f0214038cb0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.191+0000 7f021a7fc700 1 -- 172.21.15.23:0/1966684714 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f0228107080 con 0x7f0214038cb0 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 msgr2=0x7f021403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 0x7f021403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0224006fd0 tx=0x7f0224006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 msgr2=0x7f0228107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f021c009fb0 tx=0x7f021c011b20 comp rx=0 tx=0).stop 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 shutdown_connections 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f0214038cb0 0x7f021403b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 --2- 172.21.15.23:0/1966684714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02281085a0 0x7f0228107e20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 >> 172.21.15.23:0/1966684714 conn(0x7f0228101fd0 msgr2=0x7f0228102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:35.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 shutdown_connections 2024-03-16T20:26:35.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:35.194+0000 7f022f981700 1 -- 172.21.15.23:0/1966684714 wait complete. 2024-03-16T20:26:35.817 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-16T20:26:36.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:36 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:36.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:36 smithi023 ceph-mon[31879]: Added host smithi023 2024-03-16T20:26:36.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:36 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:37.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T20:26:37.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cce1b7700 1 Processor -- start 2024-03-16T20:26:37.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cce1b7700 1 -- start start 2024-03-16T20:26:37.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cce1b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:37.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cce1b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc807be30 con 0x7f2cc8108560 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36220/0 (socket says 172.21.15.23:36220) 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cc77fe700 1 -- 172.21.15.23:0/1267872184 learned_addr learned my addr 172.21.15.23:0/1267872184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cc77fe700 1 -- 172.21.15.23:0/1267872184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc810ae90 con 0x7f2cc8108560 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.250+0000 7f2cc77fe700 1 --2- 172.21.15.23:0/1267872184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80098a0 tx=0x7f2cb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9a7ccdfc88e0ef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cc67fc700 1 -- 172.21.15.23:0/1267872184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cb8004160 con 0x7f2cc8108560 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cc67fc700 1 -- 172.21.15.23:0/1267872184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cb80042c0 con 0x7f2cc8108560 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cc67fc700 1 -- 172.21.15.23:0/1267872184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cb8011ac0 con 0x7f2cc8108560 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 -- 172.21.15.23:0/1267872184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 msgr2=0x7f2cc810a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:37.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/1267872184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80098a0 tx=0x7f2cb8009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 -- 172.21.15.23:0/1267872184 shutdown_connections 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/1267872184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc810a950 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 -- 172.21.15.23:0/1267872184 >> 172.21.15.23:0/1267872184 conn(0x7f2cc8101fd0 msgr2=0x7f2cc81043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 -- 172.21.15.23:0/1267872184 shutdown_connections 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.251+0000 7f2cce1b7700 1 -- 172.21.15.23:0/1267872184 wait complete. 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cce1b7700 1 Processor -- start 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cce1b7700 1 -- start start 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cce1b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cce1b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc819be40 con 0x7f2cc8108560 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:37.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36222/0 (socket says 172.21.15.23:36222) 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.252+0000 7f2cc77fe700 1 -- 172.21.15.23:0/2728531012 learned_addr learned my addr 172.21.15.23:0/2728531012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cc77fe700 1 -- 172.21.15.23:0/2728531012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8009550 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cc77fe700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8011a70 tx=0x7f2cb8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cb80039c0 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc819c040 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cb8003b20 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cb8024610 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.253+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc819c4e0 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.254+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f2cb802d3f0 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.254+0000 7f2cc4ff9700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 0x7f2cb003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.254+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc810bcf0 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.254+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2cb804cc50 con 0x7f2cc8108560 2024-03-16T20:26:37.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.254+0000 7f2cc6ffd700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 0x7f2cb003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.255+0000 7f2cc6ffd700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 0x7f2cb003b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc006fd0 tx=0x7f2cbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.259+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2cb8010970 con 0x7f2cc8108560 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.483+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc8103270 con 0x7f2cb0038cb0 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.488+0000 7f2cc4ff9700 1 -- 172.21.15.23:0/2728531012 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2cc8103270 con 0x7f2cb0038cb0 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 msgr2=0x7f2cb003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 0x7f2cb003b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc006fd0 tx=0x7f2cbc006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 msgr2=0x7f2cc819b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8011a70 tx=0x7f2cb8011b50 comp rx=0 tx=0).stop 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 shutdown_connections 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f2cb0038cb0 0x7f2cb003b170 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 --2- 172.21.15.23:0/2728531012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cc8108560 0x7f2cc819b900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 >> 172.21.15.23:0/2728531012 conn(0x7f2cc8101fd0 msgr2=0x7f2cc8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 shutdown_connections 2024-03-16T20:26:37.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:36.490+0000 7f2cce1b7700 1 -- 172.21.15.23:0/2728531012 wait complete. 2024-03-16T20:26:37.076 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-16T20:26:37.705 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:37 smithi023 ceph-mon[31879]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:37.705 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:37 smithi023 ceph-mon[31879]: Saving service mon spec with placement count:5 2024-03-16T20:26:37.705 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:37 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:38.514 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T20:26:38.514 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.451+0000 7f7f9626b700 1 Processor -- start 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.451+0000 7f7f9626b700 1 -- start start 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f9626b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f9626b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9007bdf0 con 0x7f7f90108660 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36224/0 (socket says 172.21.15.23:36224) 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8ffff700 1 -- 172.21.15.23:0/1174148657 learned_addr learned my addr 172.21.15.23:0/1174148657 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8ffff700 1 -- 172.21.15.23:0/1174148657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9010af90 con 0x7f7f90108660 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8ffff700 1 --2- 172.21.15.23:0/1174148657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7f800098a0 tx=0x7f7f80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654311c3977bdd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8effd700 1 -- 172.21.15.23:0/1174148657 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f80004160 con 0x7f7f90108660 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8effd700 1 -- 172.21.15.23:0/1174148657 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f800042c0 con 0x7f7f90108660 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.452+0000 7f7f8effd700 1 -- 172.21.15.23:0/1174148657 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f80011ac0 con 0x7f7f90108660 2024-03-16T20:26:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 -- 172.21.15.23:0/1174148657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 msgr2=0x7f7f9010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 --2- 172.21.15.23:0/1174148657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7f800098a0 tx=0x7f7f80009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 -- 172.21.15.23:0/1174148657 shutdown_connections 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 --2- 172.21.15.23:0/1174148657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9010aa50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 -- 172.21.15.23:0/1174148657 >> 172.21.15.23:0/1174148657 conn(0x7f7f90102050 msgr2=0x7f7f90104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 -- 172.21.15.23:0/1174148657 shutdown_connections 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.453+0000 7f7f9626b700 1 -- 172.21.15.23:0/1174148657 wait complete. 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f9626b700 1 Processor -- start 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f9626b700 1 -- start start 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f9626b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f9626b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9019bee0 con 0x7f7f90108660 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36238/0 (socket says 172.21.15.23:36238) 2024-03-16T20:26:38.516 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.454+0000 7f7f8ffff700 1 -- 172.21.15.23:0/2121957156 learned_addr learned my addr 172.21.15.23:0/2121957156 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f8ffff700 1 -- 172.21.15.23:0/2121957156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80009550 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f8ffff700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7f80003f70 tx=0x7f7f80011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f800039f0 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9019c0e0 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f80003b50 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7f8001a760 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.455+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9019c580 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f7f8001a900 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9010bd90 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f8d7fa700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 0x7f7f7803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7f8004ce90 con 0x7f7f90108660 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f8f7fe700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 0x7f7f7803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:38.517 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.456+0000 7f7f8f7fe700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 0x7f7f7803b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7f84006fd0 tx=0x7f7f84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.461+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7f80018dc0 con 0x7f7f90108660 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.685+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7f90103310 con 0x7f7f78038cb0 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.690+0000 7f7f8d7fa700 1 -- 172.21.15.23:0/2121957156 <== mgr.14120 v2:172.21.15.23:6800/2899821830 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7f90103310 con 0x7f7f78038cb0 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 msgr2=0x7f7f7803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 0x7f7f7803b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7f84006fd0 tx=0x7f7f84006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 msgr2=0x7f7f9019b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7f80003f70 tx=0x7f7f80011b40 comp rx=0 tx=0).stop 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 shutdown_connections 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f7f78038cb0 0x7f7f7803b170 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 --2- 172.21.15.23:0/2121957156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f90108660 0x7f7f9019b9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 >> 172.21.15.23:0/2121957156 conn(0x7f7f90102050 msgr2=0x7f7f90102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 shutdown_connections 2024-03-16T20:26:38.518 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:37.692+0000 7f7f9626b700 1 -- 172.21.15.23:0/2121957156 wait complete. 2024-03-16T20:26:38.758 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:38 smithi023 ceph-mon[31879]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:38.759 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:38 smithi023 ceph-mon[31879]: Saving service mgr spec with placement count:2 2024-03-16T20:26:38.759 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:38 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:39.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.266+0000 7f3e54950700 1 Processor -- start 2024-03-16T20:26:39.998 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e54950700 1 -- start start 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e54950700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e54950700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50075880 con 0x7f3e50108580 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e4e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e4e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36240/0 (socket says 172.21.15.23:36240) 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e4e59c700 1 -- 172.21.15.23:0/302891199 learned_addr learned my addr 172.21.15.23:0/302891199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:40.000 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e4e59c700 1 -- 172.21.15.23:0/302891199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5010aeb0 con 0x7f3e50108580 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.267+0000 7f3e4e59c700 1 --2- 172.21.15.23:0/302891199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e400098a0 tx=0x7f3e40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3700a457ba494528 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e4d59a700 1 -- 172.21.15.23:0/302891199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40004160 con 0x7f3e50108580 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e4d59a700 1 -- 172.21.15.23:0/302891199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e400042c0 con 0x7f3e50108580 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e4d59a700 1 -- 172.21.15.23:0/302891199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40011a80 con 0x7f3e50108580 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 -- 172.21.15.23:0/302891199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 msgr2=0x7f3e5010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 --2- 172.21.15.23:0/302891199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e400098a0 tx=0x7f3e40009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 -- 172.21.15.23:0/302891199 shutdown_connections 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 --2- 172.21.15.23:0/302891199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5010a970 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 -- 172.21.15.23:0/302891199 >> 172.21.15.23:0/302891199 conn(0x7f3e50101fb0 msgr2=0x7f3e501043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 -- 172.21.15.23:0/302891199 shutdown_connections 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.268+0000 7f3e54950700 1 -- 172.21.15.23:0/302891199 wait complete. 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e54950700 1 Processor -- start 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e54950700 1 -- start start 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e54950700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e54950700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5019bdf0 con 0x7f3e50108580 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e4e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e4e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36244/0 (socket says 172.21.15.23:36244) 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.269+0000 7f3e4e59c700 1 -- 172.21.15.23:0/499093062 learned_addr learned my addr 172.21.15.23:0/499093062 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e4e59c700 1 -- 172.21.15.23:0/499093062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40009550 con 0x7f3e50108580 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e4e59c700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009130 tx=0x7f3e40011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40003aa0 con 0x7f3e50108580 2024-03-16T20:26:40.002 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5019bff0 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e40003c00 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e400248a0 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.270+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5019c490 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.271+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f3e40024ae0 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.271+0000 7f3e477fe700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 0x7f3e3c03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.271+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3e4004df00 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.271+0000 7f3e4dd9b700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 0x7f3e3c03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.271+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5010bcf0 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.272+0000 7f3e4dd9b700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 0x7f3e3c03b170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e38006fd0 tx=0x7f3e38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.276+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3e40024d90 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.483+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f3e5010bed0 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.485+0000 7f3e477fe700 1 -- 172.21.15.23:0/499093062 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3e4004b030 con 0x7f3e50108580 2024-03-16T20:26:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 msgr2=0x7f3e3c03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 0x7f3e3c03b170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e38006fd0 tx=0x7f3e38006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 msgr2=0x7f3e5019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009130 tx=0x7f3e40011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 shutdown_connections 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f3e3c038cb0 0x7f3e3c03b170 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.487+0000 7f3e54950700 1 --2- 172.21.15.23:0/499093062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e50108580 0x7f3e5019b8b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.488+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 >> 172.21.15.23:0/499093062 conn(0x7f3e50101fb0 msgr2=0x7f3e50102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.488+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 shutdown_connections 2024-03-16T20:26:40.004 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:39.488+0000 7f3e54950700 1 -- 172.21.15.23:0/499093062 wait complete. 2024-03-16T20:26:40.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:39 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:40.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:39 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:40.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:39 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/499093062' entity='client.admin' 2024-03-16T20:26:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.392+0000 7fca48170700 1 Processor -- start 2024-03-16T20:26:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.392+0000 7fca48170700 1 -- start start 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.392+0000 7fca48170700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.392+0000 7fca48170700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca401073a0 con 0x7fca401069b0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca45f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca45f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36252/0 (socket says 172.21.15.23:36252) 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca45f0c700 1 -- 172.21.15.23:0/3021027113 learned_addr learned my addr 172.21.15.23:0/3021027113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca45f0c700 1 -- 172.21.15.23:0/3021027113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca40107bb0 con 0x7fca401069b0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca45f0c700 1 --2- 172.21.15.23:0/3021027113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0098a0 tx=0x7fca3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9d2944478f1b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca44f0a700 1 -- 172.21.15.23:0/3021027113 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca3c004160 con 0x7fca401069b0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca44f0a700 1 -- 172.21.15.23:0/3021027113 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca3c0042c0 con 0x7fca401069b0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.393+0000 7fca44f0a700 1 -- 172.21.15.23:0/3021027113 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca3c011a80 con 0x7fca401069b0 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- 172.21.15.23:0/3021027113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 msgr2=0x7fca40106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 --2- 172.21.15.23:0/3021027113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0098a0 tx=0x7fca3c009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- 172.21.15.23:0/3021027113 shutdown_connections 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 --2- 172.21.15.23:0/3021027113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40106dd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- 172.21.15.23:0/3021027113 >> 172.21.15.23:0/3021027113 conn(0x7fca40101fb0 msgr2=0x7fca401043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- 172.21.15.23:0/3021027113 shutdown_connections 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- 172.21.15.23:0/3021027113 wait complete. 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 Processor -- start 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.394+0000 7fca48170700 1 -- start start 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca48170700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca48170700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400759c0 con 0x7fca401069b0 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca45f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca45f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36254/0 (socket says 172.21.15.23:36254) 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca45f0c700 1 -- 172.21.15.23:0/1598192165 learned_addr learned my addr 172.21.15.23:0/1598192165 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca45f0c700 1 -- 172.21.15.23:0/1598192165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009550 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca45f0c700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca3c0039e0 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4019c430 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca3c003b40 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca3c024820 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.395+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca401a0cf0 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.396+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fca3c024980 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.396+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4010ade0 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.396+0000 7fca36ffd700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 0x7fca2c03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.396+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fca3c04cf30 con 0x7fca401069b0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.396+0000 7fca4570b700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 0x7fca2c03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:41.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.397+0000 7fca4570b700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 0x7fca2c03b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca30006fd0 tx=0x7fca30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.401+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fca3c010380 con 0x7fca401069b0 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.617+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fca4010aff0 con 0x7fca401069b0 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.619+0000 7fca36ffd700 1 -- 172.21.15.23:0/1598192165 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fca3c016b10 con 0x7fca401069b0 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 msgr2=0x7fca2c03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 0x7fca2c03b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fca30006fd0 tx=0x7fca30006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 msgr2=0x7fca40075480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca3c000c00 tx=0x7fca3c011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 shutdown_connections 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fca2c038cb0 0x7fca2c03b170 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 --2- 172.21.15.23:0/1598192165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca401069b0 0x7fca40075480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 >> 172.21.15.23:0/1598192165 conn(0x7fca40101fb0 msgr2=0x7fca40102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 shutdown_connections 2024-03-16T20:26:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:40.622+0000 7fca48170700 1 -- 172.21.15.23:0/1598192165 wait complete. 2024-03-16T20:26:41.563 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-16T20:26:41.802 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:41 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1598192165' entity='client.admin' 2024-03-16T20:26:43.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:42 smithi023 ceph-mon[31879]: from='mgr.14120 172.21.15.23:0/723537002' entity='mgr.x' 2024-03-16T20:26:43.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:42 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/976242011' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T20:26:43.164 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:42 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: ignoring --setuser ceph since I am not root 2024-03-16T20:26:43.164 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:42 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: ignoring --setgroup ceph since I am not root 2024-03-16T20:26:43.450 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a8c9b700 1 Processor -- start 2024-03-16T20:26:43.455 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a8c9b700 1 -- start start 2024-03-16T20:26:43.455 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a8c9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:43.455 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a8c9b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4106810 con 0x7fb7a4105e20 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33984/0 (socket says 172.21.15.23:33984) 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.996+0000 7fb7a259c700 1 -- 172.21.15.23:0/2642378974 learned_addr learned my addr 172.21.15.23:0/2642378974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.997+0000 7fb7a259c700 1 -- 172.21.15.23:0/2642378974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4108db0 con 0x7fb7a4105e20 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.997+0000 7fb7a259c700 1 --2- 172.21.15.23:0/2642378974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098a0 tx=0x7fb794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65baed410f728862 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.997+0000 7fb7a159a700 1 -- 172.21.15.23:0/2642378974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb794004160 con 0x7fb7a4105e20 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.997+0000 7fb7a159a700 1 -- 172.21.15.23:0/2642378974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7940042c0 con 0x7fb7a4105e20 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.997+0000 7fb7a159a700 1 -- 172.21.15.23:0/2642378974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb794011a80 con 0x7fb7a4105e20 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/2642378974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 msgr2=0x7fb7a4106240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/2642378974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098a0 tx=0x7fb794009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/2642378974 shutdown_connections 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/2642378974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a4106240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:43.456 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/2642378974 >> 172.21.15.23:0/2642378974 conn(0x7fb7a4101790 msgr2=0x7fb7a4103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/2642378974 shutdown_connections 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.998+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/2642378974 wait complete. 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a8c9b700 1 Processor -- start 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a8c9b700 1 -- start start 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a8c9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a8c9b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a419e040 con 0x7fb7a4105e20 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33992/0 (socket says 172.21.15.23:33992) 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a259c700 1 -- 172.21.15.23:0/976242011 learned_addr learned my addr 172.21.15.23:0/976242011 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a259c700 1 -- 172.21.15.23:0/976242011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb794009550 con 0x7fb7a4105e20 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:41.999+0000 7fb7a259c700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb794011740 tx=0x7fb794011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.000+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb794003af0 con 0x7fb7a4105e20 2024-03-16T20:26:43.457 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.000+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a419e240 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.000+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79401a910 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.000+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb794024870 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.000+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a419e6e0 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fb794024b10 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a410bfe0 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7937fe700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 0x7fb78c03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb79404df80 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7a1d9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 0x7fb78c03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.001+0000 7fb7a1d9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 0x7fb78c03b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb798006fd0 tx=0x7fb798006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.006+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb794022310 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.267+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb7a41a1290 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.852+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb794018ba0 con 0x7fb7a4105e20 2024-03-16T20:26:43.458 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.854+0000 7fb7937fe700 1 -- 172.21.15.23:0/976242011 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fb79404ca30 con 0x7fb7a4105e20 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 msgr2=0x7fb78c03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 0x7fb78c03b120 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb798006fd0 tx=0x7fb798006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 msgr2=0x7fb7a419db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb794011740 tx=0x7fb794011a50 comp rx=0 tx=0).stop 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 shutdown_connections 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7fb78c038c60 0x7fb78c03b120 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 --2- 172.21.15.23:0/976242011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7a4105e20 0x7fb7a419db00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 >> 172.21.15.23:0/976242011 conn(0x7fb7a4101790 msgr2=0x7fb7a4102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 shutdown_connections 2024-03-16T20:26:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:42.856+0000 7fb7a8c9b700 1 -- 172.21.15.23:0/976242011 wait complete. 2024-03-16T20:26:43.694 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:43 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:43.436+0000 7f3d9fe98200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T20:26:43.694 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:43 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:43.536+0000 7f3d9fe98200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T20:26:44.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:43 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/976242011' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T20:26:44.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:43 smithi023 ceph-mon[31879]: mgrmap e9: x(active, since 19s) 2024-03-16T20:26:44.450 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.207+0000 7f3d9fe98200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T20:26:44.450 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.302+0000 7f3d9fe98200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T20:26:44.450 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.446+0000 7f3d9fe98200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T20:26:44.745 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.544+0000 7f3d9fe98200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T20:26:44.746 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.630+0000 7f3d9fe98200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T20:26:44.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:44.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T20:26:44.756 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.889+0000 7f9e7b940700 1 Processor -- start 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e7b940700 1 -- start start 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e7b940700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e7b940700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74107420 con 0x7f9e74106a30 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e796dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:44.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e796dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34004/0 (socket says 172.21.15.23:34004) 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e796dc700 1 -- 172.21.15.23:0/4115847538 learned_addr learned my addr 172.21.15.23:0/4115847538 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.890+0000 7f9e796dc700 1 -- 172.21.15.23:0/4115847538 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e74107c30 con 0x7f9e74106a30 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e796dc700 1 --2- 172.21.15.23:0/4115847538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800bb40 tx=0x7f9e6800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b106abb395d2a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e67fff700 1 -- 172.21.15.23:0/4115847538 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e6800dbe0 con 0x7f9e74106a30 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e67fff700 1 -- 172.21.15.23:0/4115847538 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e68004510 con 0x7f9e74106a30 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e67fff700 1 -- 172.21.15.23:0/4115847538 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e68010470 con 0x7f9e74106a30 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e7b940700 1 -- 172.21.15.23:0/4115847538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 msgr2=0x7f9e74106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.891+0000 7f9e7b940700 1 --2- 172.21.15.23:0/4115847538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800bb40 tx=0x7f9e6800be50 comp rx=0 tx=0).stop 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- 172.21.15.23:0/4115847538 shutdown_connections 2024-03-16T20:26:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 --2- 172.21.15.23:0/4115847538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74106e50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- 172.21.15.23:0/4115847538 >> 172.21.15.23:0/4115847538 conn(0x7f9e74102010 msgr2=0x7f9e74104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- 172.21.15.23:0/4115847538 shutdown_connections 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- 172.21.15.23:0/4115847538 wait complete. 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 Processor -- start 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- start start 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e7b940700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74199cd0 con 0x7f9e74106a30 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.892+0000 7f9e796dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:44.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e796dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34012/0 (socket says 172.21.15.23:34012) 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e796dc700 1 -- 172.21.15.23:0/3651940941 learned_addr learned my addr 172.21.15.23:0/3651940941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e796dc700 1 -- 172.21.15.23:0/3651940941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6800b7f0 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e796dc700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800bfd0 tx=0x7f9e680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e6801e070 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e68004260 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e74199ed0 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e68022a20 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.893+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7419a370 con 0x7f9e74106a30 2024-03-16T20:26:44.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f9e68010730 con 0x7f9e74106a30 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e741932c0 con 0x7f9e74106a30 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e667fc700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 0x7f9e6003b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9e6804f500 con 0x7f9e74106a30 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e78edb700 1 -- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 msgr2=0x7f9e6003b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.894+0000 7f9e78edb700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 0x7f9e6003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:43.899+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9e68021590 con 0x7f9e74106a30 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.095+0000 7f9e78edb700 1 -- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 msgr2=0x7f9e6003b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.095+0000 7f9e78edb700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 0x7f9e6003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T20:26:44.761 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.160+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9e7402dab0 con 0x7f9e74106a30 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.160+0000 7f9e667fc700 1 -- 172.21.15.23:0/3651940941 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f9e68027030 con 0x7f9e74106a30 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.162+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 msgr2=0x7f9e6003b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.162+0000 7f9e7b940700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 0x7f9e6003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.162+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 msgr2=0x7f9e74199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800bfd0 tx=0x7f9e680047c0 comp rx=0 tx=0).stop 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 shutdown_connections 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f9e60038d10 0x7f9e6003b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 --2- 172.21.15.23:0/3651940941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e74106a30 0x7f9e74199790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 >> 172.21.15.23:0/3651940941 conn(0x7f9e74102010 msgr2=0x7f9e74190400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:44.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 shutdown_connections 2024-03-16T20:26:44.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:44.163+0000 7f9e7b940700 1 -- 172.21.15.23:0/3651940941 wait complete. 2024-03-16T20:26:44.763 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-16T20:26:44.763 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-16T20:26:45.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3651940941' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T20:26:45.023 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:44 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:44.903+0000 7f3d9fe98200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T20:26:45.023 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:45 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:45.019+0000 7f3d9fe98200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:45.338 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:45 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:45.109+0000 7f3d9fe98200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T20:26:45.338 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:45 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:45.208+0000 7f3d9fe98200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T20:26:45.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:45 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:45.386+0000 7f3d9fe98200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T20:26:46.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:46 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:46.369+0000 7f3d9fe98200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T20:26:46.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:46 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:46.502+0000 7f3d9fe98200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T20:26:47.135 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:46 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:46.840+0000 7f3d9fe98200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T20:26:47.135 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:46 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:46.932+0000 7f3d9fe98200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T20:26:47.135 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:47 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:47.132+0000 7f3d9fe98200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:47.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:47 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:47.322+0000 7f3d9fe98200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T20:26:47.877 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:47 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:47.613+0000 7f3d9fe98200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:26:47.877 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:47 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:47.874+0000 7f3d9fe98200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T20:26:48.175 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:48 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:48.117+0000 7f3d9fe98200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T20:26:49.338 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:48 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:48.940+0000 7f3d9fe98200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T20:26:49.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:49.520+0000 7f3d9fe98200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T20:26:49.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:49.621+0000 7f3d9fe98200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T20:26:50.338 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:26:49.842+0000 7f3d9fe98200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T20:26:50.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: Active manager daemon x restarted 2024-03-16T20:26:50.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: Activating manager daemon x 2024-03-16T20:26:50.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: mgrmap e10: x(active, starting, since 0.00557427s) 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: Manager daemon x is now available 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:26:51.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-16T20:26:51.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-16T20:26:51.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:50 smithi023 ceph-mon[31879]: mgrmap e11: x(active, since 1.00616s) 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.163+0000 7f95430da700 1 Processor -- start 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f95430da700 1 -- start start 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f95430da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f95430da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c07bdf0 con 0x7f953c1085a0 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f9540e76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f9540e76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34016/0 (socket says 172.21.15.23:34016) 2024-03-16T20:26:51.463 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f9540e76700 1 -- 172.21.15.23:0/4118149080 learned_addr learned my addr 172.21.15.23:0/4118149080 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f9540e76700 1 -- 172.21.15.23:0/4118149080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c10aed0 con 0x7f953c1085a0 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f9540e76700 1 --2- 172.21.15.23:0/4118149080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f95300098a0 tx=0x7f9530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827a27f939c64a5b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.164+0000 7f953b7fe700 1 -- 172.21.15.23:0/4118149080 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9530004160 con 0x7f953c1085a0 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f953b7fe700 1 -- 172.21.15.23:0/4118149080 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95300042c0 con 0x7f953c1085a0 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f953b7fe700 1 -- 172.21.15.23:0/4118149080 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9530011a80 con 0x7f953c1085a0 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 -- 172.21.15.23:0/4118149080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 msgr2=0x7f953c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 --2- 172.21.15.23:0/4118149080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f95300098a0 tx=0x7f9530009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 -- 172.21.15.23:0/4118149080 shutdown_connections 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 --2- 172.21.15.23:0/4118149080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c10a990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 -- 172.21.15.23:0/4118149080 >> 172.21.15.23:0/4118149080 conn(0x7f953c101f70 msgr2=0x7f953c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 -- 172.21.15.23:0/4118149080 shutdown_connections 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.165+0000 7f95430da700 1 -- 172.21.15.23:0/4118149080 wait complete. 2024-03-16T20:26:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f95430da700 1 Processor -- start 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f95430da700 1 -- start start 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f95430da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f95430da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c108360 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f9540e76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f9540e76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34024/0 (socket says 172.21.15.23:34024) 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f9540e76700 1 -- 172.21.15.23:0/2424339660 learned_addr learned my addr 172.21.15.23:0/2424339660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f9540e76700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530009550 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.166+0000 7f9540e76700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9530009130 tx=0x7f9530011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.167+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9530003aa0 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.167+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9530003c00 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.167+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953c106550 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.167+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95300248a0 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.167+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953c1069f0 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f9530024a40 con 0x7f953c1085a0 2024-03-16T20:26:51.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f9539ffb700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f952c03b8d0 con 0x7f952c038d00 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f953004dea0 con 0x7f953c1085a0 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f953bfff700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.168+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.369+0000 7f953bfff700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.369+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.770+0000 7f953bfff700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:45.770+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:46.571+0000 7f953bfff700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:46.571+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:48.172+0000 7f953bfff700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2899821830 2024-03-16T20:26:51.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:48.172+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:49.860+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f953002d3f0 con 0x7f953c1085a0 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:49.860+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 msgr2=0x7f952c03b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:49.860+0000 7f9539ffb700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.860+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f9530018350 con 0x7f953c1085a0 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.860+0000 7f9539ffb700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 0x7f952c03eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.860+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f952c03b8d0 con 0x7f952c03c780 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.860+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 0x7f952c03eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.861+0000 7f953bfff700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 0x7f952c03eb70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9528003a10 tx=0x7f95280092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.862+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f952c03b8d0 con 0x7f952c03c780 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.868+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f953c0690f0 con 0x7f952c03c780 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.868+0000 7f9539ffb700 1 -- 172.21.15.23:0/2424339660 <== mgr.14152 v2:172.21.15.23:6800/3040096369 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f953c0690f0 con 0x7f952c03c780 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 msgr2=0x7f952c03eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 0x7f952c03eb70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9528003a10 tx=0x7f95280092b0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 msgr2=0x7f953c107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:51.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9530009130 tx=0x7f9530011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 shutdown_connections 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f952c03c780 0x7f952c03eb70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f953c1085a0 0x7f953c107e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 --2- 172.21.15.23:0/2424339660 >> [v2:172.21.15.23:6800/2899821830,v1:172.21.15.23:6801/2899821830] conn(0x7f952c038d00 0x7f952c03b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 >> 172.21.15.23:0/2424339660 conn(0x7f953c101f70 msgr2=0x7f953c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 shutdown_connections 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:50.869+0000 7f95430da700 1 -- 172.21.15.23:0/2424339660 wait complete. 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-16T20:26:51.468 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-16T20:26:52.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:50] ENGINE Bus STARTING 2024-03-16T20:26:52.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:50] ENGINE Serving on http://172.21.15.23:8765 2024-03-16T20:26:52.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:50] ENGINE Serving on https://172.21.15.23:7150 2024-03-16T20:26:52.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: [16/Mar/2024:20:26:50] ENGINE Bus STARTED 2024-03-16T20:26:52.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T20:26:52.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:51 smithi023 ceph-mon[31879]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T20:26:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:52 smithi023 ceph-mon[31879]: mgrmap e12: x(active, since 2s) 2024-03-16T20:26:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:26:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:52 smithi023 ceph-mon[31879]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:26:53.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:26:53.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9ccf54700 1 Processor -- start 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9ccf54700 1 -- start start 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9ccf54700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9ccf54700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c80758a0 con 0x7ff9c8108560 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53000/0 (socket says 172.21.15.23:53000) 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.157+0000 7ff9c659c700 1 -- 172.21.15.23:0/3193224883 learned_addr learned my addr 172.21.15.23:0/3193224883 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.158+0000 7ff9c659c700 1 -- 172.21.15.23:0/3193224883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c810ae90 con 0x7ff9c8108560 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.158+0000 7ff9c659c700 1 --2- 172.21.15.23:0/3193224883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80098a0 tx=0x7ff9b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aeafa2505eea18d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.158+0000 7ff9c559a700 1 -- 172.21.15.23:0/3193224883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9b8004160 con 0x7ff9c8108560 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.158+0000 7ff9c559a700 1 -- 172.21.15.23:0/3193224883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9b80042c0 con 0x7ff9c8108560 2024-03-16T20:26:53.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.158+0000 7ff9c559a700 1 -- 172.21.15.23:0/3193224883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9b8011a80 con 0x7ff9c8108560 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 -- 172.21.15.23:0/3193224883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 msgr2=0x7ff9c810a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/3193224883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80098a0 tx=0x7ff9b8009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 -- 172.21.15.23:0/3193224883 shutdown_connections 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/3193224883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c810a950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 -- 172.21.15.23:0/3193224883 >> 172.21.15.23:0/3193224883 conn(0x7ff9c8101fb0 msgr2=0x7ff9c81043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 -- 172.21.15.23:0/3193224883 shutdown_connections 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.159+0000 7ff9ccf54700 1 -- 172.21.15.23:0/3193224883 wait complete. 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9ccf54700 1 Processor -- start 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9ccf54700 1 -- start start 2024-03-16T20:26:53.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9ccf54700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9ccf54700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8108320 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53016/0 (socket says 172.21.15.23:53016) 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9c659c700 1 -- 172.21.15.23:0/1107177511 learned_addr learned my addr 172.21.15.23:0/1107177511 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9c659c700 1 -- 172.21.15.23:0/1107177511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8009550 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9c659c700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009130 tx=0x7ff9b8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.160+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9b8003aa0 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.161+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8106510 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.161+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9b8003c00 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.161+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9b80248a0 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.161+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c81069b0 con 0x7ff9c8108560 2024-03-16T20:26:53.046 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.162+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c810bcd0 con 0x7ff9c8108560 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.162+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7ff9b801ac10 con 0x7ff9c8108560 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.162+0000 7ff9b77fe700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 0x7ff9b003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.162+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff9b804cfe0 con 0x7ff9c8108560 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.162+0000 7ff9c5d9b700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 0x7ff9b003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.163+0000 7ff9c5d9b700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 0x7ff9b003b1c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc006fd0 tx=0x7ff9bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.171+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff9b8024ae0 con 0x7ff9c8108560 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.380+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9c8106fc0 con 0x7ff9b0038d00 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.466+0000 7ff9b77fe700 1 -- 172.21.15.23:0/1107177511 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff9c8106fc0 con 0x7ff9b0038d00 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 msgr2=0x7ff9b003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 0x7ff9b003b1c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc006fd0 tx=0x7ff9bc006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 msgr2=0x7ff9c8107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009130 tx=0x7ff9b8011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:53.047 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 shutdown_connections 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff9b0038d00 0x7ff9b003b1c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 --2- 172.21.15.23:0/1107177511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9c8108560 0x7ff9c8107de0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 >> 172.21.15.23:0/1107177511 conn(0x7ff9c8101fb0 msgr2=0x7ff9c8102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 shutdown_connections 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:52.468+0000 7ff9ccf54700 1 -- 172.21.15.23:0/1107177511 wait complete. 2024-03-16T20:26:53.048 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-16T20:26:54.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:26:54.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:54 smithi023 ceph-mon[31879]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:26:54.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$m9myVreJPozO.V1w6BN/9.DMGz9sadcGiS3Wfa7FSMhV67BiPXqEO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710620813, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.484+0000 7f1bc4827700 1 Processor -- start 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bc4827700 1 -- start start 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bc4827700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bc4827700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00758e0 con 0x7f1bc0108520 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bbe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bbe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53030/0 (socket says 172.21.15.23:53030) 2024-03-16T20:26:55.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bbe59c700 1 -- 172.21.15.23:0/2261237792 learned_addr learned my addr 172.21.15.23:0/2261237792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bbe59c700 1 -- 172.21.15.23:0/2261237792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc010ae50 con 0x7f1bc0108520 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.485+0000 7f1bbe59c700 1 --2- 172.21.15.23:0/2261237792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00098a0 tx=0x7f1bb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5bd6853d55d3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.486+0000 7f1bbd59a700 1 -- 172.21.15.23:0/2261237792 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb0004160 con 0x7f1bc0108520 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.486+0000 7f1bbd59a700 1 -- 172.21.15.23:0/2261237792 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bb00042c0 con 0x7f1bc0108520 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.486+0000 7f1bbd59a700 1 -- 172.21.15.23:0/2261237792 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb0011a80 con 0x7f1bc0108520 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.486+0000 7f1bc4827700 1 -- 172.21.15.23:0/2261237792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 msgr2=0x7f1bc010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.486+0000 7f1bc4827700 1 --2- 172.21.15.23:0/2261237792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00098a0 tx=0x7f1bb0009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.487+0000 7f1bc4827700 1 -- 172.21.15.23:0/2261237792 shutdown_connections 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.487+0000 7f1bc4827700 1 --2- 172.21.15.23:0/2261237792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc010a910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.487+0000 7f1bc4827700 1 -- 172.21.15.23:0/2261237792 >> 172.21.15.23:0/2261237792 conn(0x7f1bc0101fb0 msgr2=0x7f1bc01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.487+0000 7f1bc4827700 1 -- 172.21.15.23:0/2261237792 shutdown_connections 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.487+0000 7f1bc4827700 1 -- 172.21.15.23:0/2261237792 wait complete. 2024-03-16T20:26:55.282 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bc4827700 1 Processor -- start 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bc4827700 1 -- start start 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bc4827700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bc4827700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc019bdf0 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bbe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bbe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53042/0 (socket says 172.21.15.23:53042) 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bbe59c700 1 -- 172.21.15.23:0/3924939343 learned_addr learned my addr 172.21.15.23:0/3924939343 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bbe59c700 1 -- 172.21.15.23:0/3924939343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0009550 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1bbe59c700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0003f20 tx=0x7f1bb0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.488+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb00039f0 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.489+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bb0003b50 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.489+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc019bff0 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.489+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb001a6d0 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.489+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc019c490 con 0x7f1bc0108520 2024-03-16T20:26:55.283 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f1bb001a9b0 con 0x7f1bc0108520 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc010bcf0 con 0x7f1bc0108520 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1baf7fe700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 0x7f1ba803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1bb004d260 con 0x7f1bc0108520 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1bbdd9b700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 0x7f1ba803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.490+0000 7f1bbdd9b700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 0x7f1ba803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4006fd0 tx=0x7f1bb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.499+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1bb0022330 con 0x7f1bc0108520 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.718+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1bc0063430 con 0x7f1ba8038cb0 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.967+0000 7f1baf7fe700 1 -- 172.21.15.23:0/3924939343 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1bc0063430 con 0x7f1ba8038cb0 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 msgr2=0x7f1ba803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 0x7f1ba803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4006fd0 tx=0x7f1bb4006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:55.284 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 msgr2=0x7f1bc019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0003f20 tx=0x7f1bb0011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 shutdown_connections 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba8038cb0 0x7f1ba803b170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 --2- 172.21.15.23:0/3924939343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bc0108520 0x7f1bc019b8b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 >> 172.21.15.23:0/3924939343 conn(0x7f1bc0101fb0 msgr2=0x7f1bc0102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 shutdown_connections 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:53.969+0000 7f1bc4827700 1 -- 172.21.15.23:0/3924939343 wait complete. 2024-03-16T20:26:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-16T20:26:56.179 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2534303749' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabea1f4700 1 Processor -- start 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabea1f4700 1 -- start start 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabea1f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabea1f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe407bdf0 con 0x7fabe41085a0 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53046/0 (socket says 172.21.15.23:53046) 2024-03-16T20:26:56.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe37fe700 1 -- 172.21.15.23:0/2163515855 learned_addr learned my addr 172.21.15.23:0/2163515855 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe37fe700 1 -- 172.21.15.23:0/2163515855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe410aed0 con 0x7fabe41085a0 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe37fe700 1 --2- 172.21.15.23:0/2163515855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabd40098a0 tx=0x7fabd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cd2c87d71bf178 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.667+0000 7fabe27fc700 1 -- 172.21.15.23:0/2163515855 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd4004160 con 0x7fabe41085a0 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabe27fc700 1 -- 172.21.15.23:0/2163515855 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd40042c0 con 0x7fabe41085a0 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabe27fc700 1 -- 172.21.15.23:0/2163515855 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd4011a80 con 0x7fabe41085a0 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 -- 172.21.15.23:0/2163515855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 msgr2=0x7fabe410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2163515855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabd40098a0 tx=0x7fabd4009bb0 comp rx=0 tx=0).stop 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 -- 172.21.15.23:0/2163515855 shutdown_connections 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2163515855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe410a990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 -- 172.21.15.23:0/2163515855 >> 172.21.15.23:0/2163515855 conn(0x7fabe4101fd0 msgr2=0x7fabe41043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 -- 172.21.15.23:0/2163515855 shutdown_connections 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.668+0000 7fabea1f4700 1 -- 172.21.15.23:0/2163515855 wait complete. 2024-03-16T20:26:56.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabea1f4700 1 Processor -- start 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabea1f4700 1 -- start start 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabea1f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabea1f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe419bde0 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabe37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabe37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53058/0 (socket says 172.21.15.23:53058) 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabe37fe700 1 -- 172.21.15.23:0/2534303749 learned_addr learned my addr 172.21.15.23:0/2534303749 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabe37fe700 1 -- 172.21.15.23:0/2534303749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4009550 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.669+0000 7fabe37fe700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabd4003f20 tx=0x7fabd4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd40039f0 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd4003b50 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe419bfe0 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd401a6d0 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe419c480 con 0x7fabe41085a0 2024-03-16T20:26:56.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.670+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fabd401a9c0 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.671+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe410bd10 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.671+0000 7fabe0ff9700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 0x7fabcc03b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.671+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fabd404d090 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.671+0000 7fabe2ffd700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 0x7fabcc03b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.671+0000 7fabe2ffd700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 0x7fabcc03b1d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fabd8006fd0 tx=0x7fabd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.676+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fabd4022740 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.878+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fabe419ed60 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.878+0000 7fabe0ff9700 1 -- 172.21.15.23:0/2534303749 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fabd402c350 con 0x7fabe41085a0 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 msgr2=0x7fabcc03b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 0x7fabcc03b1d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fabd8006fd0 tx=0x7fabd8006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 msgr2=0x7fabe419b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabd4003f20 tx=0x7fabd4011b00 comp rx=0 tx=0).stop 2024-03-16T20:26:56.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 shutdown_connections 2024-03-16T20:26:56.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fabcc038d10 0x7fabcc03b1d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:56.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 --2- 172.21.15.23:0/2534303749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabe41085a0 0x7fabe419b8a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:56.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 >> 172.21.15.23:0/2534303749 conn(0x7fabe4101fd0 msgr2=0x7fabe4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:56.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 shutdown_connections 2024-03-16T20:26:56.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:55.880+0000 7fabea1f4700 1 -- 172.21.15.23:0/2534303749 wait complete. 2024-03-16T20:26:56.451 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T20:26:56.451 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-16T20:26:56.458 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T20:26:56.458 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-16T20:26:56.458 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-16T20:26:56.459 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T20:26:56.460 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-16T20:26:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-16T20:26:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-16T20:26:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: Password: z5t56qoark 2024-03-16T20:26:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:56.461 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config directory 2024-03-16T20:26:56.461 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-16T20:26:57.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:56 smithi023 ceph-mon[31879]: mgrmap e13: x(active, since 6s) 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef9141700 1 Processor -- start 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef9141700 1 -- start start 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef9141700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef9141700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef407bdf0 con 0x7f6ef41085a0 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.866+0000 7f6ef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53072/0 (socket says 172.21.15.23:53072) 2024-03-16T20:26:57.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef2d9d700 1 -- 172.21.15.23:0/1106485819 learned_addr learned my addr 172.21.15.23:0/1106485819 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef2d9d700 1 -- 172.21.15.23:0/1106485819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef410aed0 con 0x7f6ef41085a0 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef2d9d700 1 --2- 172.21.15.23:0/1106485819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e80 tx=0x7f6ee400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8593002555934e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef1d9b700 1 -- 172.21.15.23:0/1106485819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ee400e070 con 0x7f6ef41085a0 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef1d9b700 1 -- 172.21.15.23:0/1106485819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ee400b8c0 con 0x7f6ef41085a0 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.867+0000 7f6ef1d9b700 1 -- 172.21.15.23:0/1106485819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ee40044a0 con 0x7f6ef41085a0 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 -- 172.21.15.23:0/1106485819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 msgr2=0x7f6ef410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1106485819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e80 tx=0x7f6ee400b330 comp rx=0 tx=0).stop 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 -- 172.21.15.23:0/1106485819 shutdown_connections 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1106485819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef410a990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 -- 172.21.15.23:0/1106485819 >> 172.21.15.23:0/1106485819 conn(0x7f6ef4101fd0 msgr2=0x7f6ef41043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 -- 172.21.15.23:0/1106485819 shutdown_connections 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 -- 172.21.15.23:0/1106485819 wait complete. 2024-03-16T20:26:57.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.868+0000 7f6ef9141700 1 Processor -- start 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef9141700 1 -- start start 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef9141700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef9141700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef4108360 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53076/0 (socket says 172.21.15.23:53076) 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef2d9d700 1 -- 172.21.15.23:0/1304676057 learned_addr learned my addr 172.21.15.23:0/1304676057 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef2d9d700 1 -- 172.21.15.23:0/1304676057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4009b30 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.869+0000 7f6ef2d9d700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4004070 tx=0x7f6ee400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ee400e040 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ee4003e60 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef4106550 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ee401c7d0 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef4106a50 con 0x7f6ef41085a0 2024-03-16T20:26:57.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6ee401ca40 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.870+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef410bd10 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.871+0000 7f6eebfff700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 0x7f6ee003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.871+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6ee404ce00 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.871+0000 7f6ef259c700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 0x7f6ee003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.871+0000 7f6ef259c700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 0x7f6ee003b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:56.876+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6ee4011960 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.079+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6ef410bf20 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.080+0000 7f6eebfff700 1 -- 172.21.15.23:0/1304676057 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f6ee40200c0 con 0x7f6ef41085a0 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 msgr2=0x7f6ee003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 0x7f6ee003b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 msgr2=0x7f6ef4107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4004070 tx=0x7f6ee400bd80 comp rx=0 tx=0).stop 2024-03-16T20:26:57.646 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 shutdown_connections 2024-03-16T20:26:57.647 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6ee0038db0 0x7f6ee003b270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:57.647 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 --2- 172.21.15.23:0/1304676057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef41085a0 0x7f6ef4107e20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:57.647 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 >> 172.21.15.23:0/1304676057 conn(0x7f6ef4101fd0 msgr2=0x7f6ef4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:57.647 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 shutdown_connections 2024-03-16T20:26:57.647 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:57.082+0000 7f6ef9141700 1 -- 172.21.15.23:0/1304676057 wait complete. 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face9f1c700 1 Processor -- start 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face9f1c700 1 -- start start 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face9f1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face9f1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face407be30 con 0x7face4108610 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:58.991 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53090/0 (socket says 172.21.15.23:53090) 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.071+0000 7face37fe700 1 -- 172.21.15.23:0/3151514748 learned_addr learned my addr 172.21.15.23:0/3151514748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.072+0000 7face37fe700 1 -- 172.21.15.23:0/3151514748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face410af40 con 0x7face4108610 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.072+0000 7face37fe700 1 --2- 172.21.15.23:0/3151514748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facd4009e80 tx=0x7facd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ceca4c514b92153 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.072+0000 7face27fc700 1 -- 172.21.15.23:0/3151514748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd400e070 con 0x7face4108610 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.072+0000 7face27fc700 1 -- 172.21.15.23:0/3151514748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facd400b8c0 con 0x7face4108610 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.072+0000 7face27fc700 1 -- 172.21.15.23:0/3151514748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd40044a0 con 0x7face4108610 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 -- 172.21.15.23:0/3151514748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 msgr2=0x7face410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 --2- 172.21.15.23:0/3151514748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facd4009e80 tx=0x7facd400b330 comp rx=0 tx=0).stop 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 -- 172.21.15.23:0/3151514748 shutdown_connections 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 --2- 172.21.15.23:0/3151514748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face410aa00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 -- 172.21.15.23:0/3151514748 >> 172.21.15.23:0/3151514748 conn(0x7face4102020 msgr2=0x7face4104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 -- 172.21.15.23:0/3151514748 shutdown_connections 2024-03-16T20:26:58.992 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 -- 172.21.15.23:0/3151514748 wait complete. 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.073+0000 7face9f1c700 1 Processor -- start 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face9f1c700 1 -- start start 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face9f1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face9f1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41083d0 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53098/0 (socket says 172.21.15.23:53098) 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face37fe700 1 -- 172.21.15.23:0/3264382105 learned_addr learned my addr 172.21.15.23:0/3264382105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face37fe700 1 -- 172.21.15.23:0/3264382105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009b30 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face37fe700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facd4004250 tx=0x7facd400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd400e040 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facd4003de0 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd401c7d0 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.074+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face41065c0 con 0x7face4108610 2024-03-16T20:26:58.993 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.075+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4106ac0 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.075+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7facd401ca40 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.076+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4063610 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.076+0000 7face0ff9700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 0x7facd003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.076+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7facd404cf00 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.076+0000 7face2ffd700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 0x7facd003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.076+0000 7face2ffd700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 0x7facd003b270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faccc006fd0 tx=0x7faccc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.081+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7facd40118f0 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.414+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7face410bde0 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.422+0000 7face0ff9700 1 -- 172.21.15.23:0/3264382105 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7facd4017070 con 0x7face4108610 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 msgr2=0x7facd003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 0x7facd003b270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faccc006fd0 tx=0x7faccc006e40 comp rx=0 tx=0).stop 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 msgr2=0x7face4107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:26:58.994 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7facd4004250 tx=0x7facd400bd80 comp rx=0 tx=0).stop 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 shutdown_connections 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7facd0038db0 0x7facd003b270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 --2- 172.21.15.23:0/3264382105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7face4108610 0x7face4107e90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 >> 172.21.15.23:0/3264382105 conn(0x7face4102020 msgr2=0x7face4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 shutdown_connections 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-16T20:26:58.424+0000 7face9f1c700 1 -- 172.21.15.23:0/3264382105 wait complete. 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T20:26:58.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:26:58.996 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-16T20:26:59.029 INFO:tasks.cephadm:Fetching config... 2024-03-16T20:26:59.030 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:26:59.030 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T20:26:59.045 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T20:26:59.045 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:26:59.045 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T20:26:59.101 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T20:26:59.101 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:26:59.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T20:26:59.171 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T20:26:59.171 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:26:59.171 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T20:26:59.225 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T20:26:59.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T20:26:59.304 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:59.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T20:26:59.376 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:59.394 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T20:26:59.443 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCX+Hf69nWj/1sBhxJ2gKNBvlsS+G66GSCLR3J1KnL4qsrymIGtwL5gca465kXtf3SlUesjVYQIAHbJipyUKCw81fMmZXpOP+kncbd5cecTDaKhUf9634NTvyRG/oy9jRApXvvEdmUxfryReFtZfzH2CwLEjuWBoYN3KLh6r7tgS/D5gwzgxcCcOFLN2dyLVGKphjVik9dwxSRKbRWYAoUbYLDsjFMk61jOZI957YiC7tvzS7/eH8aIA8dtF0Svlqjk6S1caoQesX7JWunhNvejtXTU3nKb1+Zd6lPSnSrDmzyP9uSpXrNG0bK1m4BJyV7RJwnrSgv8//Ewlgy7ZADVscSALSTNNrI/wgXUBN26SMz5RHxkwl4DsxfUFMXk4FBzoXXKlaXyLI+dcDUq26a0rAb3NVzo8Xz9C4RUKTmMHSfrpcuFaHjp2I2PqtBZmgtJdmFLAb+nOi6EikupdnszmykS/264mFR1/pg3OJU+2Ja3JrxFmOsfV700kOcJOpM= ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:26:59.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T20:26:59.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:26:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3264382105' entity='client.admin' 2024-03-16T20:26:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 -- 172.21.15.23:0/3339885468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 msgr2=0x7fba4c1048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 --2- 172.21.15.23:0/3339885468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c1048b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009910 tx=0x7fba3c009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 -- 172.21.15.23:0/3339885468 shutdown_connections 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 --2- 172.21.15.23:0/3339885468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c1048b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 -- 172.21.15.23:0/3339885468 >> 172.21.15.23:0/3339885468 conn(0x7fba4c0fbef0 msgr2=0x7fba4c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 -- 172.21.15.23:0/3339885468 shutdown_connections 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.306+0000 7fba508ef700 1 -- 172.21.15.23:0/3339885468 wait complete. 2024-03-16T20:27:01.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.307+0000 7fba508ef700 1 Processor -- start 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.307+0000 7fba508ef700 1 -- start start 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.307+0000 7fba508ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.307+0000 7fba508ef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c102230 con 0x7fba4c1024c0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba4a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba4a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49796/0 (socket says 172.21.15.23:49796) 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba4a59c700 1 -- 172.21.15.23:0/3932022793 learned_addr learned my addr 172.21.15.23:0/3932022793 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba4a59c700 1 -- 172.21.15.23:0/3932022793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c0095f0 con 0x7fba4c1024c0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba4a59c700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba3c004f40 tx=0x7fba3c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba3c01d070 con 0x7fba4c1024c0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba3c00be40 con 0x7fba4c1024c0 2024-03-16T20:27:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c100470 con 0x7fba4c1024c0 2024-03-16T20:27:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba3c00f460 con 0x7fba4c1024c0 2024-03-16T20:27:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.308+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c100910 con 0x7fba4c1024c0 2024-03-16T20:27:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.309+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fba3c00f650 con 0x7fba4c1024c0 2024-03-16T20:27:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.309+0000 7fba3b7fe700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 0x7fba3403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.309+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fba3c022470 con 0x7fba4c1024c0 2024-03-16T20:27:01.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.309+0000 7fba49d9b700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 0x7fba3403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:01.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.309+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c1081c0 con 0x7fba4c1024c0 2024-03-16T20:27:01.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.310+0000 7fba49d9b700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 0x7fba3403b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba40006fd0 tx=0x7fba40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:01.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.314+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba3c027020 con 0x7fba4c1024c0 2024-03-16T20:27:01.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.516+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fba4c1078c0 con 0x7fba4c1024c0 2024-03-16T20:27:01.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.519+0000 7fba3b7fe700 1 -- 172.21.15.23:0/3932022793 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fba3c02a360 con 0x7fba4c1024c0 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 msgr2=0x7fba3403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 0x7fba3403b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba40006fd0 tx=0x7fba40006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 msgr2=0x7fba4c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba3c004f40 tx=0x7fba3c005020 comp rx=0 tx=0).stop 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 shutdown_connections 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fba34038db0 0x7fba3403b270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:01.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 --2- 172.21.15.23:0/3932022793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba4c1024c0 0x7fba4c101cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:01.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 >> 172.21.15.23:0/3932022793 conn(0x7fba4c0fbef0 msgr2=0x7fba4c0fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:01.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 shutdown_connections 2024-03-16T20:27:01.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:01.521+0000 7fba508ef700 1 -- 172.21.15.23:0/3932022793 wait complete. 2024-03-16T20:27:02.371 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T20:27:02.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T20:27:02.610 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:02.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3932022793' entity='client.admin' 2024-03-16T20:27:02.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:02.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:02.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:27:02.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:02.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:02.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.059+0000 7f74fdec0700 1 -- 172.21.15.23:0/3224638528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 msgr2=0x7f74f81048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.059+0000 7f74fdec0700 1 --2- 172.21.15.23:0/3224638528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f81048d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009910 tx=0x7f74e0009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.059+0000 7f74fdec0700 1 -- 172.21.15.23:0/3224638528 shutdown_connections 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.059+0000 7f74fdec0700 1 --2- 172.21.15.23:0/3224638528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f81048d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.059+0000 7f74fdec0700 1 -- 172.21.15.23:0/3224638528 >> 172.21.15.23:0/3224638528 conn(0x7f74f80fbeb0 msgr2=0x7f74f80fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 -- 172.21.15.23:0/3224638528 shutdown_connections 2024-03-16T20:27:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 -- 172.21.15.23:0/3224638528 wait complete. 2024-03-16T20:27:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 Processor -- start 2024-03-16T20:27:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 -- start start 2024-03-16T20:27:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.060+0000 7f74fdec0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f8197420 con 0x7f74f81024e0 2024-03-16T20:27:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49806/0 (socket says 172.21.15.23:49806) 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f77fe700 1 -- 172.21.15.23:0/1337281920 learned_addr learned my addr 172.21.15.23:0/1337281920 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f77fe700 1 -- 172.21.15.23:0/1337281920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00095f0 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f77fe700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f74e0004f40 tx=0x7f74e0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74e001d070 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f8197620 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74e000be40 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74e000f460 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.061+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f8197ac0 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.062+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f74e000f6a0 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.062+0000 7f74f4ff9700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 0x7f74e403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.062+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f74e004d030 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.062+0000 7f74f6ffd700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 0x7f74e403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.062+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f81908b0 con 0x7f74f81024e0 2024-03-16T20:27:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.063+0000 7f74f6ffd700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 0x7f74e403b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74e8006fd0 tx=0x7f74e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:04.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.067+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f74e0027020 con 0x7f74f81024e0 2024-03-16T20:27:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.281+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f74f8062420 con 0x7f74e4038d50 2024-03-16T20:27:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.284+0000 7f74f4ff9700 1 -- 172.21.15.23:0/1337281920 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f74f8062420 con 0x7f74e4038d50 2024-03-16T20:27:04.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 msgr2=0x7f74e403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:04.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 0x7f74e403b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74e8006fd0 tx=0x7f74e8006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:04.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 msgr2=0x7f74f8196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f74e0004f40 tx=0x7f74e0005020 comp rx=0 tx=0).stop 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 shutdown_connections 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f74e4038d50 0x7f74e403b210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 --2- 172.21.15.23:0/1337281920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f81024e0 0x7f74f8196ee0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.286+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 >> 172.21.15.23:0/1337281920 conn(0x7f74f80fbeb0 msgr2=0x7f74f80fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.287+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 shutdown_connections 2024-03-16T20:27:04.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:04.287+0000 7f74fdec0700 1 -- 172.21.15.23:0/1337281920 wait complete. 2024-03-16T20:27:05.031 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi031 2024-03-16T20:27:05.031 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:27:05.032 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/etc/ceph/ceph.conf 2024-03-16T20:27:05.053 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:27:05.053 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:05.115 INFO:tasks.cephadm:Adding host smithi031 to orchestrator... 2024-03-16T20:27:05.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch host add smithi031 2024-03-16T20:27:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: Updating smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:05.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:05.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:05.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:05.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:06.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:06 smithi023 ceph-mon[31879]: Updating smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T20:27:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2203786193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 msgr2=0x7f1bb0106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2203786193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb0106c80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009910 tx=0x7f1ba4009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2203786193 shutdown_connections 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2203786193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb0106c80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2203786193 >> 172.21.15.23:0/2203786193 conn(0x7f1bb00fc390 msgr2=0x7f1bb00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2203786193 shutdown_connections 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.708+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2203786193 wait complete. 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb6c19700 1 Processor -- start 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb6c19700 1 -- start start 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb6c19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb6c19700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0199a00 con 0x7f1bb01068a0 2024-03-16T20:27:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb49b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb49b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49834/0 (socket says 172.21.15.23:49834) 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb49b5700 1 -- 172.21.15.23:0/2456538146 learned_addr learned my addr 172.21.15.23:0/2456538146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb49b5700 1 -- 172.21.15.23:0/2456538146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba40095f0 con 0x7f1bb01068a0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.709+0000 7f1bb49b5700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004f40 tx=0x7f1ba4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.710+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba401d070 con 0x7f1bb01068a0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.710+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ba400bbd0 con 0x7f1bb01068a0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.710+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb0199c60 con 0x7f1bb01068a0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.710+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ba4003900 con 0x7f1bb01068a0 2024-03-16T20:27:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.710+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb019a0e0 con 0x7f1bb01068a0 2024-03-16T20:27:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f1ba4003b70 con 0x7f1bb01068a0 2024-03-16T20:27:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb0063610 con 0x7f1bb01068a0 2024-03-16T20:27:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1badffb700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 0x7f1ba003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ba404cc30 con 0x7f1bb01068a0 2024-03-16T20:27:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1baffff700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 0x7f1ba003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.711+0000 7f1baffff700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 0x7f1ba003b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c006fd0 tx=0x7f1b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.716+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ba4029950 con 0x7f1bb01068a0 2024-03-16T20:27:06.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:06.924+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bb010ae90 con 0x7f1ba0038db0 2024-03-16T20:27:08.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:07 smithi023 ceph-mon[31879]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:08.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:07 smithi023 ceph-mon[31879]: Deploying cephadm binary to smithi031 2024-03-16T20:27:11.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.330+0000 7f1badffb700 1 -- 172.21.15.23:0/2456538146 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1bb010ae90 con 0x7f1ba0038db0 2024-03-16T20:27:11.334 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi031' with addr '172.21.15.31' 2024-03-16T20:27:11.334 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:10 smithi023 ceph-mon[31879]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 msgr2=0x7f1ba003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 0x7f1ba003b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c006fd0 tx=0x7f1b9c006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 msgr2=0x7f1bb01994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004f40 tx=0x7f1ba4005020 comp rx=0 tx=0).stop 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 shutdown_connections 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ba0038db0 0x7f1ba003b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.332+0000 7f1bb6c19700 1 --2- 172.21.15.23:0/2456538146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bb01068a0 0x7f1bb01994c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.333+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 >> 172.21.15.23:0/2456538146 conn(0x7f1bb00fc390 msgr2=0x7f1bb0069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:11.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.333+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 shutdown_connections 2024-03-16T20:27:11.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:11.333+0000 7f1bb6c19700 1 -- 172.21.15.23:0/2456538146 wait complete. 2024-03-16T20:27:12.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T20:27:12.461 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:12.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:12.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:12 smithi023 ceph-mon[31879]: Added host smithi031 2024-03-16T20:27:12.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:12.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:12 smithi023 ceph-mon[31879]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:12.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:13.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:13 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:14.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 -- 172.21.15.23:0/831776529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 msgr2=0x7f7808107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 --2- 172.21.15.23:0/831776529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808107da0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7804009910 tx=0x7f7804009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 -- 172.21.15.23:0/831776529 shutdown_connections 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 --2- 172.21.15.23:0/831776529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808107da0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 -- 172.21.15.23:0/831776529 >> 172.21.15.23:0/831776529 conn(0x7f78080fd4f0 msgr2=0x7f78080ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 -- 172.21.15.23:0/831776529 shutdown_connections 2024-03-16T20:27:14.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.033+0000 7f78100ae700 1 -- 172.21.15.23:0/831776529 wait complete. 2024-03-16T20:27:14.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f78100ae700 1 Processor -- start 2024-03-16T20:27:14.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f78100ae700 1 -- start start 2024-03-16T20:27:14.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f78100ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:14.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f78100ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78081978a0 con 0x7f78081079c0 2024-03-16T20:27:14.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f780de4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f780de4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39016/0 (socket says 172.21.15.23:39016) 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.034+0000 7f780de4a700 1 -- 172.21.15.23:0/3030711018 learned_addr learned my addr 172.21.15.23:0/3030711018 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f780de4a700 1 -- 172.21.15.23:0/3030711018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78040095f0 con 0x7f78081079c0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f780de4a700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7804004f40 tx=0x7f7804005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f780401d070 con 0x7f78081079c0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f780400bbd0 con 0x7f78081079c0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7808197b00 con 0x7f78081079c0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7804003900 con 0x7f78081079c0 2024-03-16T20:27:14.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.035+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7808197f80 con 0x7f78081079c0 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f7804003b70 con 0x7f78081079c0 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7808103100 con 0x7f78081079c0 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f77feffd700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 0x7f77f403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f780404cb10 con 0x7f78081079c0 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f780d649700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 0x7f77f403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:14.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.036+0000 7f780d649700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 0x7f77f403b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:14.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.041+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7804029540 con 0x7f78081079c0 2024-03-16T20:27:14.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.250+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f780810be70 con 0x7f77f4038db0 2024-03-16T20:27:14.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.251+0000 7f77feffd700 1 -- 172.21.15.23:0/3030711018 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f780810be70 con 0x7f77f4038db0 2024-03-16T20:27:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:27:14.254 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}] 2024-03-16T20:27:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 msgr2=0x7f77f403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 0x7f77f403b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 msgr2=0x7f7808197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7804004f40 tx=0x7f7804005020 comp rx=0 tx=0).stop 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 shutdown_connections 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f77f4038db0 0x7f77f403b270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 --2- 172.21.15.23:0/3030711018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78081079c0 0x7f7808197360 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.253+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 >> 172.21.15.23:0/3030711018 conn(0x7f78080fd4f0 msgr2=0x7f7808109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.254+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 shutdown_connections 2024-03-16T20:27:14.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:14.254+0000 7f78100ae700 1 -- 172.21.15.23:0/3030711018 wait complete. 2024-03-16T20:27:14.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:14 smithi023 ceph-mon[31879]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:14.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:14.805 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-03-16T20:27:14.805 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:27:14.805 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-03-16T20:27:14.824 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:27:14.824 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:14.881 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-03-16T20:27:14.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch host add smithi189 2024-03-16T20:27:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:15.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:15 smithi023 ceph-mon[31879]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:27:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 -- 172.21.15.23:0/4039311367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 msgr2=0x7f338c107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 --2- 172.21.15.23:0/4039311367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c107da0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3374009910 tx=0x7f3374009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 -- 172.21.15.23:0/4039311367 shutdown_connections 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 --2- 172.21.15.23:0/4039311367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c107da0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 -- 172.21.15.23:0/4039311367 >> 172.21.15.23:0/4039311367 conn(0x7f338c0fd4f0 msgr2=0x7f338c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 -- 172.21.15.23:0/4039311367 shutdown_connections 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.553+0000 7f3391d46700 1 -- 172.21.15.23:0/4039311367 wait complete. 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.554+0000 7f3391d46700 1 Processor -- start 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.554+0000 7f3391d46700 1 -- start start 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.554+0000 7f3391d46700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.554+0000 7f3391d46700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c197850 con 0x7f338c1079c0 2024-03-16T20:27:16.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:16 smithi023 ceph-mon[31879]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f338b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39038/0 (socket says 172.21.15.23:39038) 2024-03-16T20:27:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f338b7fe700 1 -- 172.21.15.23:0/1901970048 learned_addr learned my addr 172.21.15.23:0/1901970048 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f338b7fe700 1 -- 172.21.15.23:0/1901970048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740095f0 con 0x7f338c1079c0 2024-03-16T20:27:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f338b7fe700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3374004f40 tx=0x7f3374005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f337401d070 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f337400bbd0 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3374003900 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c197ab0 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.555+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c197f30 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.556+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f3374003bc0 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.556+0000 7f3388ff9700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 0x7f337803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.556+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f337404cad0 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.557+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c063610 con 0x7f338c1079c0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.557+0000 7f338affd700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 0x7f337803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.557+0000 7f338affd700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 0x7f337803b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f337c006fd0 tx=0x7f337c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:16.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.562+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3374026080 con 0x7f338c1079c0 2024-03-16T20:27:16.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:16.771+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7f338c10be00 con 0x7f3378038d50 2024-03-16T20:27:17.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:17.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:17.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:17.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:17.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:17.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:27:17.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:17.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: Deploying cephadm binary to smithi189 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:18.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:18 smithi023 ceph-mon[31879]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T20:27:20.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:19 smithi023 ceph-mon[31879]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:21.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.031+0000 7f3388ff9700 1 -- 172.21.15.23:0/1901970048 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f338c10be00 con 0x7f3378038d50 2024-03-16T20:27:21.034 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-03-16T20:27:21.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 msgr2=0x7f337803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 0x7f337803b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f337c006fd0 tx=0x7f337c006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 msgr2=0x7f338c197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3374004f40 tx=0x7f3374005020 comp rx=0 tx=0).stop 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 shutdown_connections 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3378038d50 0x7f337803b210 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 --2- 172.21.15.23:0/1901970048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c1079c0 0x7f338c197310 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.033+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 >> 172.21.15.23:0/1901970048 conn(0x7f338c0fd4f0 msgr2=0x7f338c1094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.034+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 shutdown_connections 2024-03-16T20:27:21.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:21.034+0000 7f3391d46700 1 -- 172.21.15.23:0/1901970048 wait complete. 2024-03-16T20:27:21.597 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:21.597 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:21.597 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:21.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:21.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:21.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T20:27:21.868 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:22.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:22 smithi023 ceph-mon[31879]: Added host smithi189 2024-03-16T20:27:22.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:22.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:22 smithi023 ceph-mon[31879]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:22.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:23.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.169+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510639655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 msgr2=0x7fc7cc106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:23.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.169+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510639655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc106c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009910 tx=0x7fc7bc009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.169+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510639655 shutdown_connections 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.169+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510639655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc106c90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.169+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510639655 >> 172.21.15.23:0/1510639655 conn(0x7fc7cc0fc3c0 msgr2=0x7fc7cc0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510639655 shutdown_connections 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510639655 wait complete. 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 Processor -- start 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 -- start start 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.170+0000 7fc7d2052700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc199a60 con 0x7fc7cc1068b0 2024-03-16T20:27:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7cb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7cb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52516/0 (socket says 172.21.15.23:52516) 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7cb7fe700 1 -- 172.21.15.23:0/1510189758 learned_addr learned my addr 172.21.15.23:0/1510189758 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7cb7fe700 1 -- 172.21.15.23:0/1510189758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0095f0 con 0x7fc7cc1068b0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7cb7fe700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc004f40 tx=0x7fc7bc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7bc01d070 con 0x7fc7cc1068b0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7bc00bbd0 con 0x7fc7cc1068b0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc199cc0 con 0x7fc7cc1068b0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7bc003900 con 0x7fc7cc1068b0 2024-03-16T20:27:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.171+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc19a140 con 0x7fc7cc1068b0 2024-03-16T20:27:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.172+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fc7bc003bc0 con 0x7fc7cc1068b0 2024-03-16T20:27:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.172+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc063610 con 0x7fc7cc1068b0 2024-03-16T20:27:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.172+0000 7fc7c8ff9700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 0x7fc7b403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.172+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc7bc04cbe0 con 0x7fc7cc1068b0 2024-03-16T20:27:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.173+0000 7fc7caffd700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 0x7fc7b403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.173+0000 7fc7caffd700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 0x7fc7b403b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0006fd0 tx=0x7fc7c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.177+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc7bc02ae20 con 0x7fc7cc1068b0 2024-03-16T20:27:23.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.385+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7cc10aea0 con 0x7fc7b4038d50 2024-03-16T20:27:23.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.387+0000 7fc7c8ff9700 1 -- 172.21.15.23:0/1510189758 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7fc7cc10aea0 con 0x7fc7b4038d50 2024-03-16T20:27:23.390 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:27:23.390 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.388+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 msgr2=0x7fc7b403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 0x7fc7b403b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0006fd0 tx=0x7fc7c0006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 msgr2=0x7fc7cc199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc004f40 tx=0x7fc7bc005020 comp rx=0 tx=0).stop 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 shutdown_connections 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc7b4038d50 0x7fc7b403b210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 --2- 172.21.15.23:0/1510189758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7cc1068b0 0x7fc7cc199520 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 >> 172.21.15.23:0/1510189758 conn(0x7fc7cc0fc3c0 msgr2=0x7fc7cc069ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 shutdown_connections 2024-03-16T20:27:23.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:23.389+0000 7fc7d2052700 1 -- 172.21.15.23:0/1510189758 wait complete. 2024-03-16T20:27:23.909 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T20:27:23.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T20:27:24.149 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:24.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:24.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:24 smithi023 ceph-mon[31879]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:27:24.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:24 smithi023 ceph-mon[31879]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 -- 172.21.15.23:0/1984211091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 msgr2=0x7f87fc106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 --2- 172.21.15.23:0/1984211091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc106c80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009910 tx=0x7f87ec009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 -- 172.21.15.23:0/1984211091 shutdown_connections 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 --2- 172.21.15.23:0/1984211091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc106c80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 -- 172.21.15.23:0/1984211091 >> 172.21.15.23:0/1984211091 conn(0x7f87fc0fc390 msgr2=0x7f87fc0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 -- 172.21.15.23:0/1984211091 shutdown_connections 2024-03-16T20:27:25.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.432+0000 7f880263d700 1 -- 172.21.15.23:0/1984211091 wait complete. 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f880263d700 1 Processor -- start 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f880263d700 1 -- start start 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f880263d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f880263d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc197830 con 0x7f87fc1068a0 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f87fbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f87fbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52532/0 (socket says 172.21.15.23:52532) 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f87fbfff700 1 -- 172.21.15.23:0/4157345777 learned_addr learned my addr 172.21.15.23:0/4157345777 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:25.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f87fbfff700 1 -- 172.21.15.23:0/4157345777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec0095f0 con 0x7f87fc1068a0 2024-03-16T20:27:25.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.433+0000 7f87fbfff700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f87ec004f40 tx=0x7f87ec005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:25.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.434+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87ec01d070 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.434+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc197a90 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.434+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87ec00bbd0 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.434+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87ec003900 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.434+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc197f10 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f87ec003bc0 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc063610 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f87f97fa700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 0x7f87e403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f87ec04cab0 con 0x7f87fc1068a0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f87fb7fe700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 0x7f87e403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:25.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.435+0000 7f87fb7fe700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 0x7f87e403b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:25.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.440+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f87ec029910 con 0x7f87fc1068a0 2024-03-16T20:27:25.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.650+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f87fc10ae20 con 0x7f87fc1068a0 2024-03-16T20:27:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.702+0000 7f87f97fa700 1 -- 172.21.15.23:0/4157345777 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f87ec026020 con 0x7f87fc1068a0 2024-03-16T20:27:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 msgr2=0x7f87e403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 0x7f87e403b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 msgr2=0x7f87fc1972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f87ec004f40 tx=0x7f87ec005020 comp rx=0 tx=0).stop 2024-03-16T20:27:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 shutdown_connections 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f87e4038d50 0x7f87e403b210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 --2- 172.21.15.23:0/4157345777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87fc1068a0 0x7f87fc1972f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 >> 172.21.15.23:0/4157345777 conn(0x7f87fc0fc390 msgr2=0x7f87fc1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 shutdown_connections 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:25.704+0000 7f880263d700 1 -- 172.21.15.23:0/4157345777 wait complete. 2024-03-16T20:27:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-16T20:27:25.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:25 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4157345777' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T20:27:26.253 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-16T20:27:26.253 INFO:tasks.cephadm:Adding mon.b on smithi031 2024-03-16T20:27:26.253 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-03-16T20:27:26.253 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c' 2024-03-16T20:27:26.493 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T20:27:26.540 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-16T20:27:27.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4157345777' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T20:27:27.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:27.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 -- 172.21.15.189:0/717514747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 msgr2=0x7faef4106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 --2- 172.21.15.189:0/717514747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef4106ca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faee4009910 tx=0x7faee4009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 -- 172.21.15.189:0/717514747 shutdown_connections 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 --2- 172.21.15.189:0/717514747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef4106ca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 -- 172.21.15.189:0/717514747 >> 172.21.15.189:0/717514747 conn(0x7faef40fc3b0 msgr2=0x7faef40fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 -- 172.21.15.189:0/717514747 shutdown_connections 2024-03-16T20:27:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 -- 172.21.15.189:0/717514747 wait complete. 2024-03-16T20:27:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.040+0000 7faef90d1700 1 Processor -- start 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef90d1700 1 -- start start 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef90d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef90d1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef41977e0 con 0x7faef41068c0 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.189:34188/0 (socket says 172.21.15.189:34188) 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef2d9d700 1 -- 172.21.15.189:0/2309429922 learned_addr learned my addr 172.21.15.189:0/2309429922 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef2d9d700 1 -- 172.21.15.189:0/2309429922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40095f0 con 0x7faef41068c0 2024-03-16T20:27:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.041+0000 7faef2d9d700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faee4004f40 tx=0x7faee4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faee401d070 con 0x7faef41068c0 2024-03-16T20:27:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef4197a40 con 0x7faef41068c0 2024-03-16T20:27:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faee400bbd0 con 0x7faef41068c0 2024-03-16T20:27:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faee4003900 con 0x7faef41068c0 2024-03-16T20:27:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef4197ec0 con 0x7faef41068c0 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.042+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7faee4003bc0 con 0x7faef41068c0 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.043+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef4063610 con 0x7faef41068c0 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.043+0000 7faedbfff700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 0x7faedc03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.043+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faee404cc10 con 0x7faef41068c0 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.043+0000 7faef259c700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 0x7faedc03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.043+0000 7faef259c700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 0x7faedc03b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faee8006fd0 tx=0x7faee8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:28.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.052+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faee4029540 con 0x7faef41068c0 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:28.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:28.260 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.256+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}) v1 -- 0x7faef410aeb0 con 0x7faedc038d50 2024-03-16T20:27:28.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.264+0000 7faedbfff700 1 -- 172.21.15.189:0/2309429922 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faef410aeb0 con 0x7faedc038d50 2024-03-16T20:27:28.267 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-03-16T20:27:28.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.265+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 msgr2=0x7faedc03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:28.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.265+0000 7faef90d1700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 0x7faedc03b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faee8006fd0 tx=0x7faee8006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:28.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.265+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 msgr2=0x7faef41972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.265+0000 7faef90d1700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faee4004f40 tx=0x7faee4005020 comp rx=0 tx=0).stop 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 shutdown_connections 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faedc038d50 0x7faedc03b210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 --2- 172.21.15.189:0/2309429922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faef41068c0 0x7faef41972a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 >> 172.21.15.189:0/2309429922 conn(0x7faef40fc3b0 msgr2=0x7faef4108330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 shutdown_connections 2024-03-16T20:27:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:28.266+0000 7faef90d1700 1 -- 172.21.15.189:0/2309429922 wait complete. 2024-03-16T20:27:28.816 DEBUG:teuthology.orchestra.run.smithi031:mon.b> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T20:27:28.819 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T20:27:28.821 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T20:27:28.821 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T20:27:29.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:29.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:30.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:29 smithi023 ceph-mon[31879]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:30.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:29 smithi023 ceph-mon[31879]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c;count:3 2024-03-16T20:27:30.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:29 smithi023 ceph-mon[31879]: Deploying daemon mon.c on smithi189 2024-03-16T20:27:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:27:31.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:30 smithi023 ceph-mon[31879]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:32.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:31 smithi023 ceph-mon[31879]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- 172.21.15.189:0/3351619137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 msgr2=0x7fdd3c101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 --2- 172.21.15.189:0/3351619137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c101790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400b0a0 tx=0x7fdd3400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- 172.21.15.189:0/3351619137 shutdown_connections 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 --2- 172.21.15.189:0/3351619137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c101790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- 172.21.15.189:0/3351619137 >> 172.21.15.189:0/3351619137 conn(0x7fdd3c0fc0f0 msgr2=0x7fdd3c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- 172.21.15.189:0/3351619137 shutdown_connections 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- 172.21.15.189:0/3351619137 wait complete. 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 Processor -- start 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.684+0000 7fdd43398700 1 -- start start 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd43398700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd43398700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1979b0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.689 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd41134700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:32.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd41134700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.189:55530/0 (socket says 172.21.15.189:55530) 2024-03-16T20:27:32.690 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd41134700 1 -- 172.21.15.189:0/602681190 learned_addr learned my addr 172.21.15.189:0/602681190 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:27:32.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd41134700 1 -- 172.21.15.189:0/602681190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34009d00 con 0x7fdd3c1013b0 2024-03-16T20:27:32.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.685+0000 7fdd41134700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400b830 tx=0x7fdd3400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:32.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.686+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd3400e040 con 0x7fdd3c1013b0 2024-03-16T20:27:32.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.686+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c197bb0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.686+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd34003c10 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.686+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd3401d6b0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.686+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c197fd0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fdd34019070 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c102160 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd327fc700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 0x7fdd2803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd3404cdf0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd40933700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 0x7fdd2803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:32.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.687+0000 7fdd40933700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 0x7fdd2803b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c006fd0 tx=0x7fdd2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:32.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.693+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdd3401c5b0 con 0x7fdd3c1013b0 2024-03-16T20:27:32.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.978+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd3c10ae20 con 0x7fdd3c1013b0 2024-03-16T20:27:32.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.979+0000 7fdd327fc700 1 -- 172.21.15.189:0/602681190 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fdd34017020 con 0x7fdd3c1013b0 2024-03-16T20:27:32.983 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:27:32.983 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","modified":"2024-03-16T20:25:43.452836Z","created":"2024-03-16T20:25:43.452836Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 msgr2=0x7fdd2803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 0x7fdd2803b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c006fd0 tx=0x7fdd2c006e40 comp rx=0 tx=0).stop 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 msgr2=0x7fdd3c197470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400b830 tx=0x7fdd3400bd40 comp rx=0 tx=0).stop 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 shutdown_connections 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fdd28038d50 0x7fdd2803b210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 --2- 172.21.15.189:0/602681190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd3c1013b0 0x7fdd3c197470 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 >> 172.21.15.189:0/602681190 conn(0x7fdd3c0fc0f0 msgr2=0x7fdd3c100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 shutdown_connections 2024-03-16T20:27:32.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:32.981+0000 7fdd43398700 1 -- 172.21.15.189:0/602681190 wait complete. 2024-03-16T20:27:32.986 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-03-16T20:27:33.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 podman[36601]: 2024-03-16 20:27:33.041648712 +0000 UTC m=+0.540674100 container start 11bcfd600bc11020d9a274576b361e414bff4d4c1f6ae381249cc4ab68da5ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:27:33.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T20:27:33.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev), process ceph-mon, pid 2 2024-03-16T20:27:33.068 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pidfile_write: ignore empty --pid-file 2024-03-16T20:27:33.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:33 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/602681190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: load: jerasure load: lrc 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: RocksDB version: 7.9.2 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Git sha 0 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Compile date 2024-03-15 02:27:09 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: DB SUMMARY 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: DB Session ID: C6K50QIRL4D0UY0753V6 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: CURRENT file: CURRENT 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T20:27:33.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.error_if_exists: 0 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.create_if_missing: 0 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.paranoid_checks: 1 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.env: 0x5627155abae0 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.info_log: 0x562717bd5900 2024-03-16T20:27:33.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.statistics: (nil) 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.use_fsync: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_log_file_size: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_fallocate: 1 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.use_direct_reads: 0 2024-03-16T20:27:33.396 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.db_log_dir: 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.wal_dir: 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.write_buffer_manager: 0x562717ba9400 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T20:27:33.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.unordered_write: 0 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T20:27:33.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.row_cache: None 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.wal_filter: None 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.two_write_queues: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.wal_compression: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.atomic_flush: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T20:27:33.399 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.log_readahead_size: 0 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_background_jobs: 2 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_background_compactions: -1 2024-03-16T20:27:33.400 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_subcompactions: 1 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_open_files: -1 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T20:27:33.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_background_flushes: -1 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Compression algorithms supported: 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kZSTD supported: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kXpressCompression supported: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kZlibCompression supported: 1 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kSnappyCompression supported: 1 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kLZ4Compression supported: 1 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: kBZip2Compression supported: 0 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T20:27:33.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.merge_operator: 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_filter: None 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562717bd5a00) 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T20:27:33.404 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_type: 0 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_index_type: 0 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_shortening: 1 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: checksum: 4 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: no_block_cache: 0 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache: 0x562717bd9090 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_options: 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: capacity : 536870912 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_shard_bits : 4 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: strict_capacity_limit : 0 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_compressed: (nil) 2024-03-16T20:27:33.405 INFO:journalctl@ceph.mon.c.smithi189.stdout: persistent_cache: (nil) 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size: 4096 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size_deviation: 10 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_restart_interval: 16 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_block_restart_interval: 1 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: metadata_block_size: 4096 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: partition_filters: 0 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: use_delta_encoding: 1 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: filter_policy: bloomfilter 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: whole_key_filtering: 1 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: verify_compression: 0 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: format_version: 5 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_index_compression: 1 2024-03-16T20:27:33.406 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_align: 0 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout: max_auto_readahead_size: 262144 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout: prepopulate_block_cache: 0 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout: initial_auto_readahead_size: 8192 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression: NoCompression 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.num_levels: 7 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T20:27:33.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T20:27:33.408 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T20:27:33.409 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T20:27:33.410 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T20:27:33.411 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.table_properties_collectors: 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.inplace_update_support: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.bloom_locality: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.max_successive_merges: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T20:27:33.412 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.ttl: 2592000 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enable_blob_files: false 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.min_blob_size: 0 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T20:27:33.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f70aacb0-f96b-4b5b-8bf3-9543fcafd9c8 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620853071092, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620853071532, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710620853, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f70aacb0-f96b-4b5b-8bf3-9543fcafd9c8", "db_session_id": "C6K50QIRL4D0UY0753V6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710620853071612, "job": 1, "event": "recovery_finished"} 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562717cca000 2024-03-16T20:27:33.414 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: DB pointer 0x562717cb4000 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** DB Stats ** 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T20:27:33.415 INFO:journalctl@ceph.mon.c.smithi189.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T20:27:33.416 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache BinnedLRUCache@0x562717bd9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: starting mon.c rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:27:33.417 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(???) e0 preinit fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: e1 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: legacy client fscid: -1 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout: No filesystems configured 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T20:27:33.418 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:27:33.419 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi031:/etc/ceph/ceph.conf 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Deploying cephadm binary to smithi189 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:33.420 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Added host smithi189 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4157345777' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4157345777' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.421 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:33.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c;count:3 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: Deploying daemon mon.c on smithi189 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/602681190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 ceph-mon[36655]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 bash[36601]: 11bcfd600bc11020d9a274576b361e414bff4d4c1f6ae381249cc4ab68da5ece 2024-03-16T20:27:33.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:33 smithi189 systemd[1]: Started Ceph mon.c for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:27:34.680 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T20:27:34.680 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T20:27:34.928 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:27:38.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: mon.a calling monitor election 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: mon.c calling monitor election 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T20:27:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: fsmap 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: mgrmap e13: x(active, since 48s) 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: overall HEALTH_OK 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:38.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: mon.a calling monitor election 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: mon.c calling monitor election 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: fsmap 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:38.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: mgrmap e13: x(active, since 48s) 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: overall HEALTH_OK 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:38.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:39.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 -- 172.21.15.189:0/3080213238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b380035c0 msgr2=0x7f0b38005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:39.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 --2- 172.21.15.189:0/3080213238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b380035c0 0x7f0b38005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c06a200 tx=0x7f0b4800feb0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 -- 172.21.15.189:0/3080213238 shutdown_connections 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 --2- 172.21.15.189:0/3080213238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b380035c0 0x7f0b38005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 -- 172.21.15.189:0/3080213238 >> 172.21.15.189:0/3080213238 conn(0x7f0b4c0fc4f0 msgr2=0x7f0b4c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 -- 172.21.15.189:0/3080213238 shutdown_connections 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.301+0000 7f0b53445700 1 -- 172.21.15.189:0/3080213238 wait complete. 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 Processor -- start 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 -- start start 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 0x7f0b4c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c102890 con 0x7f0b4c100ee0 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b53445700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c1a35c0 con 0x7f0b380035c0 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:35650/0 (socket says 172.21.15.189:35650) 2024-03-16T20:27:39.306 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 -- 172.21.15.189:0/1850230258 learned_addr learned my addr 172.21.15.189:0/1850230258 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:27:39.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 -- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 msgr2=0x7f0b4c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T20:27:39.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 -- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 msgr2=0x7f0b4c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T20:27:39.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T20:27:39.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b511e1700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:27:39.307 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.302+0000 7f0b509e0700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 0x7f0b4c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:39.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b509e0700 1 -- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 msgr2=0x7f0b4c1009a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:27:39.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b509e0700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b509e0700 1 -- 172.21.15.189:0/1850230258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4800f900 con 0x7f0b4c100ee0 2024-03-16T20:27:39.308 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b509e0700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 0x7f0b4c101360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00b4d0 tx=0x7f0b3c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0b3c011630 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b3c011c70 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0b3c010de0 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.303+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c1a3760 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.304+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c1a3c80 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.305+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f0b3c011790 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.305+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c191310 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.305+0000 7f0b427fc700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 0x7f0b3003afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.305+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0b3c04b440 con 0x7f0b4c100ee0 2024-03-16T20:27:39.309 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.305+0000 7f0b511e1700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 0x7f0b3003afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:39.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.306+0000 7f0b511e1700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 0x7f0b3003afb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b48001eb0 tx=0x7f0b48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:39.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.311+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0b3c0179b0 con 0x7f0b4c100ee0 2024-03-16T20:27:39.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:39 smithi189 ceph-mon[36655]: Deploying daemon mon.b on smithi031 2024-03-16T20:27:39.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:39.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:39 smithi023 ceph-mon[31879]: Deploying daemon mon.b on smithi031 2024-03-16T20:27:39.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:39.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.586+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b4c109a40 con 0x7f0b4c100ee0 2024-03-16T20:27:39.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.587+0000 7f0b427fc700 1 -- 172.21.15.189:0/1850230258 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f0b3c0172c0 con 0x7f0b4c100ee0 2024-03-16T20:27:39.590 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:27:39.590 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","modified":"2024-03-16T20:27:33.095069Z","created":"2024-03-16T20:25:43.452836Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 msgr2=0x7f0b3003afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 0x7f0b3003afb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b48001eb0 tx=0x7f0b48005960 comp rx=0 tx=0).stop 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 msgr2=0x7f0b4c101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 0x7f0b4c101360 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00b4d0 tx=0x7f0b3c00b890 comp rx=0 tx=0).stop 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 shutdown_connections 2024-03-16T20:27:39.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b30038af0 0x7f0b3003afb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b380035c0 0x7f0b4c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 --2- 172.21.15.189:0/1850230258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b4c100ee0 0x7f0b4c101360 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 >> 172.21.15.189:0/1850230258 conn(0x7f0b4c0fc4f0 msgr2=0x7f0b4c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 shutdown_connections 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:39.589+0000 7f0b53445700 1 -- 172.21.15.189:0/1850230258 wait complete. 2024-03-16T20:27:39.593 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-16T20:27:40.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:40 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/1850230258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:40.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:40 smithi189 ceph-mon[36655]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:40.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:40 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/1850230258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:40.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:40 smithi023 ceph-mon[31879]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:41.108 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T20:27:41.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T20:27:41.345 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:27:42.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:41 smithi189 ceph-mon[36655]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:42.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:41 smithi023 ceph-mon[31879]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:42.436 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:42 smithi031 systemd[1]: Started Ceph mon.b for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:27:47.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 -- 172.21.15.189:0/3995419414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f40057d0 msgr2=0x7f15f4007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:47.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 --2- 172.21.15.189:0/3995419414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f40057d0 0x7f15f4007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f160c069950 tx=0x7f160000fe60 comp rx=0 tx=0).stop 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 -- 172.21.15.189:0/3995419414 shutdown_connections 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 --2- 172.21.15.189:0/3995419414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f40057d0 0x7f15f4007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 -- 172.21.15.189:0/3995419414 >> 172.21.15.189:0/3995419414 conn(0x7f160c0770d0 msgr2=0x7f160c0774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 -- 172.21.15.189:0/3995419414 shutdown_connections 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.165+0000 7f16110ba700 1 -- 172.21.15.189:0/3995419414 wait complete. 2024-03-16T20:27:47.170 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 Processor -- start 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 -- start start 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 0x7f160c19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c19a370 con 0x7f160c199c80 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c1a62c0 con 0x7f160c19e550 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f16110ba700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c1a6400 con 0x7f15f40057d0 2024-03-16T20:27:47.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.167+0000 7f160ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 0x7f160c19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39228/0 (socket says 172.21.15.189:39228) 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 -- 172.21.15.189:0/2657704560 learned_addr learned my addr 172.21.15.189:0/2657704560 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160b59e700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 msgr2=0x7f160c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 msgr2=0x7f160c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160b59e700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 msgr2=0x7f160c19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160b59e700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 msgr2=0x7f160c19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160b59e700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160ad9d700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160b59e700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-16T20:27:47.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 msgr2=0x7f160c199740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 msgr2=0x7f160c19ea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 -- 172.21.15.189:0/2657704560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160000f8b0 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f160a59c700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 0x7f160c19e010 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00b550 tx=0x7f15fc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc004230 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15fc004390 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c1a66e0 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.168+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc011b90 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.169+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c1a6c30 con 0x7f160c199c80 2024-03-16T20:27:47.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.169+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f15fc011cf0 con 0x7f160c199c80 2024-03-16T20:27:47.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.170+0000 7f15f3fff700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 0x7f15f803b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:47.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.170+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c105430 con 0x7f160c199c80 2024-03-16T20:27:47.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.170+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15fc04c470 con 0x7f160c199c80 2024-03-16T20:27:47.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.170+0000 7f160ad9d700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 0x7f15f803b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:47.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.170+0000 7f160ad9d700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 0x7f15f803b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1600005850 tx=0x7f1600001e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:47.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.175+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f15fc01a400 con 0x7f160c199c80 2024-03-16T20:27:47.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.465+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f160c107860 con 0x7f160c199c80 2024-03-16T20:27:47.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: mon.c calling monitor election 2024-03-16T20:27:47.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:27:47.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: mon.a calling monitor election 2024-03-16T20:27:47.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: fsmap 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: mgrmap e13: x(active, since 57s) 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: mon.b (rank 2) addr [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] is down (out of quorum) 2024-03-16T20:27:47.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.466+0000 7f15f3fff700 1 -- 172.21.15.189:0/2657704560 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1279 (secure 0 0 0) 0x7f15fc00c150 con 0x7f160c199c80 2024-03-16T20:27:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-16T20:27:47.472 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","modified":"2024-03-16T20:27:42.043976Z","created":"2024-03-16T20:25:43.452836Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 msgr2=0x7f15f803b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 0x7f15f803b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1600005850 tx=0x7f1600001e50 comp rx=0 tx=0).stop 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 msgr2=0x7f160c19e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 0x7f160c19e010 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00b550 tx=0x7f15fc00b910 comp rx=0 tx=0).stop 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 shutdown_connections 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f15f80391e0 0x7f15f803b6a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f160c19e550 0x7f160c19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15f40057d0 0x7f160c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 --2- 172.21.15.189:0/2657704560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f160c199c80 0x7f160c19e010 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:47.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 >> 172.21.15.189:0/2657704560 conn(0x7f160c0770d0 msgr2=0x7f160c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:47.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 shutdown_connections 2024-03-16T20:27:47.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:47.468+0000 7f16110ba700 1 -- 172.21.15.189:0/2657704560 wait complete. 2024-03-16T20:27:47.474 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: mon.c calling monitor election 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: mon.a calling monitor election 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: fsmap 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: mgrmap e13: x(active, since 57s) 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: mon.b (rank 2) addr [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] is down (out of quorum) 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:47.979 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T20:27:47.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T20:27:48.234 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:48.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:48.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:48 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/2657704560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:48.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:48 smithi023 ceph-mon[31879]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:48.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:48.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:48.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:48 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/2657704560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:48.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:48 smithi189 ceph-mon[36655]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:48.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.c calling monitor election 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.a calling monitor election 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:27:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: fsmap 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mgrmap e13: x(active, since 57s) 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.b (rank 2) addr [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] is down (out of quorum) 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/2657704560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:49.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:49.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 -- 172.21.15.23:0/1374777290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 msgr2=0x7feafc106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:49.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 --2- 172.21.15.23:0/1374777290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc106c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009910 tx=0x7feaf8009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 -- 172.21.15.23:0/1374777290 shutdown_connections 2024-03-16T20:27:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 --2- 172.21.15.23:0/1374777290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc106c80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 -- 172.21.15.23:0/1374777290 >> 172.21.15.23:0/1374777290 conn(0x7feafc0fc390 msgr2=0x7feafc0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 -- 172.21.15.23:0/1374777290 shutdown_connections 2024-03-16T20:27:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.826+0000 7feb04183700 1 -- 172.21.15.23:0/1374777290 wait complete. 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 Processor -- start 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 -- start start 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 0x7feafc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feafc19e520 0x7feafc19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19a390 con 0x7feafc1068a0 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc1a6290 con 0x7feafc19e520 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb04183700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc1a63d0 con 0x7feafc199ca0 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb01f1f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:49.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb01f1f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55328/0 (socket says 172.21.15.23:55328) 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb01f1f700 1 -- 172.21.15.23:0/1815066257 learned_addr learned my addr 172.21.15.23:0/1815066257 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb02720700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feafc19e520 0x7feafc19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.827+0000 7feb0171e700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 0x7feafc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 -- 172.21.15.23:0/1815066257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 msgr2=0x7feafc19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 0x7feafc19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 -- 172.21.15.23:0/1815066257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feafc19e520 msgr2=0x7feafc19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feafc19e520 0x7feafc19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 -- 172.21.15.23:0/1815066257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80095f0 con 0x7feafc1068a0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb0171e700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 0x7feafc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb01f1f700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feaf8005b40 tx=0x7feaf800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf801d070 con 0x7feafc1068a0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc1a66b0 con 0x7feafc1068a0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf800fb50 con 0x7feafc1068a0 2024-03-16T20:27:49.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc1a6bb0 con 0x7feafc1068a0 2024-03-16T20:27:49.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.828+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8022c50 con 0x7feafc1068a0 2024-03-16T20:27:49.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.829+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7feaf802c440 con 0x7feafc1068a0 2024-03-16T20:27:49.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.829+0000 7feaf2ffd700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 0x7feae803b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:49.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.829+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc063610 con 0x7feafc1068a0 2024-03-16T20:27:49.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.829+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feaf804cd50 con 0x7feafc1068a0 2024-03-16T20:27:49.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.830+0000 7feb0171e700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 0x7feae803b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:49.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.830+0000 7feb0171e700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 0x7feae803b6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feaec009910 tx=0x7feaec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:49.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:49.835+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7feaf8027070 con 0x7feafc1068a0 2024-03-16T20:27:49.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mon.b calling monitor election 2024-03-16T20:27:49.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mon.b calling monitor election 2024-03-16T20:27:49.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mon.c calling monitor election 2024-03-16T20:27:49.940 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mon.a calling monitor election 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: fsmap 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: mgrmap e13: x(active, since 59s) 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: overall HEALTH_OK 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.941 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:49.942 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:49.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mon.b calling monitor election 2024-03-16T20:27:49.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mon.b calling monitor election 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mon.c calling monitor election 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mon.a calling monitor election 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: fsmap 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: mgrmap e13: x(active, since 59s) 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: overall HEALTH_OK 2024-03-16T20:27:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:49.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:49.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:49.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:50.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.043+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7feafc10ac90 con 0x7feafc1068a0 2024-03-16T20:27:50.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.043+0000 7feaf2ffd700 1 -- 172.21.15.23:0/1815066257 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7feaf802a740 con 0x7feafc1068a0 2024-03-16T20:27:50.047 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:27:50.047 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-16T20:27:50.047 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:27:50.047 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 msgr2=0x7feae803b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 0x7feae803b6a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feaec009910 tx=0x7feaec008040 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 msgr2=0x7feafc199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feaf8005b40 tx=0x7feaf800bcf0 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 shutdown_connections 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feae80391e0 0x7feae803b6a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feafc1068a0 0x7feafc199760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feafc19e520 0x7feafc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 --2- 172.21.15.23:0/1815066257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feafc199ca0 0x7feafc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 >> 172.21.15.23:0/1815066257 conn(0x7feafc0fc390 msgr2=0x7feafc069ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 shutdown_connections 2024-03-16T20:27:50.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:27:50.045+0000 7feb04183700 1 -- 172.21.15.23:0/1815066257 wait complete. 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.b calling monitor election 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.b calling monitor election 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.c calling monitor election 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.a calling monitor election 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: fsmap 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T20:27:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: mgrmap e13: x(active, since 59s) 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: overall HEALTH_OK 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:50.632 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T20:27:50.632 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:27:50.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T20:27:50.661 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:27:50.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:50.728 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:27:50.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T20:27:50.766 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:27:50.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:50.839 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:27:50.839 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T20:27:50.874 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:27:50.874 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T20:27:50.945 INFO:tasks.cephadm:Adding mgr.x on smithi023 2024-03-16T20:27:50.945 INFO:tasks.cephadm:Adding mgr.y on smithi031 2024-03-16T20:27:50.945 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-03-16T20:27:50.945 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi023=x;smithi031=y;smithi189=z' 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: Updating smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: Reconfiguring daemon mon.a on smithi023 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:50.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:50.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1815066257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:50.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:51.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: Updating smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T20:27:51.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: Reconfiguring daemon mon.a on smithi023 2024-03-16T20:27:51.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:51.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:51.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1815066257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:51.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:51.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: Updating smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: Updating smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: Updating smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: Reconfiguring daemon mon.a on smithi023 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1815066257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:51.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:27:51.205 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 -- 172.21.15.189:0/3397395040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74103230 msgr2=0x7f6b74103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 --2- 172.21.15.189:0/3397395040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74103230 0x7f6b74103650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009910 tx=0x7f6b5c009c20 comp rx=0 tx=0).stop 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 -- 172.21.15.189:0/3397395040 shutdown_connections 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 --2- 172.21.15.189:0/3397395040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74103230 0x7f6b74103650 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 -- 172.21.15.189:0/3397395040 >> 172.21.15.189:0/3397395040 conn(0x7f6b740fd0d0 msgr2=0x7f6b740ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 -- 172.21.15.189:0/3397395040 shutdown_connections 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.588+0000 7f6b78d13700 1 -- 172.21.15.189:0/3397395040 wait complete. 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 Processor -- start 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 -- start start 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b74103230 0x7f6b74197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74197c40 0x7f6b7419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:52.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419f560 con 0x7f6b74197c40 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419f6d0 con 0x7f6b74103230 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b78d13700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7419f890 con 0x7f6b741977c0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57822/0 (socket says 172.21.15.189:57822) 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 -- 172.21.15.189:0/1194412874 learned_addr learned my addr 172.21.15.189:0/1194412874 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b72d9d700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74197c40 0x7f6b7419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b7259c700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b74103230 0x7f6b74197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 -- 172.21.15.189:0/1194412874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b74103230 msgr2=0x7f6b74197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b74103230 0x7f6b74197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 -- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74197c40 msgr2=0x7f6b7419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74197c40 0x7f6b7419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 -- 172.21.15.189:0/1194412874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0095f0 con 0x7f6b741977c0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.589+0000 7f6b71d9b700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400b780 tx=0x7f6b6400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64004230 con 0x7f6b741977c0 2024-03-16T20:27:52.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b64004390 con 0x7f6b741977c0 2024-03-16T20:27:52.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7419fb70 con 0x7f6b741977c0 2024-03-16T20:27:52.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64011b90 con 0x7f6b741977c0 2024-03-16T20:27:52.595 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b741a00f0 con 0x7f6b741977c0 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.590+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6b64011cf0 con 0x7f6b741977c0 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.591+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b74190a60 con 0x7f6b741977c0 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.591+0000 7f6b6b7fe700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 0x7f6b6003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.591+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b6404b390 con 0x7f6b741977c0 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.591+0000 7f6b7259c700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 0x7f6b6003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:27:52.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.591+0000 7f6b7259c700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 0x7f6b6003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c004690 tx=0x7f6b5c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:27:52.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.596+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6b6401ae40 con 0x7f6b741977c0 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: Reconfiguring mon.b (monmap changed)... 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: Reconfiguring daemon mon.b on smithi031 2024-03-16T20:27:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:52 smithi031 ceph-mon[36281]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: Reconfiguring mon.b (monmap changed)... 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:52.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: Reconfiguring daemon mon.b on smithi031 2024-03-16T20:27:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:52 smithi189 ceph-mon[36655]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:52.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.799+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi031=y;smithi189=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b7402d990 con 0x7f6b600391e0 2024-03-16T20:27:52.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.807+0000 7f6b6b7fe700 1 -- 172.21.15.189:0/1194412874 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6b7402d990 con 0x7f6b600391e0 2024-03-16T20:27:52.810 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-03-16T20:27:52.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 msgr2=0x7f6b6003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:52.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 0x7f6b6003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c004690 tx=0x7f6b5c00b540 comp rx=0 tx=0).stop 2024-03-16T20:27:52.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 msgr2=0x7f6b7419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:27:52.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400b780 tx=0x7f6b6400ba90 comp rx=0 tx=0).stop 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 shutdown_connections 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6b600391e0 0x7f6b6003b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b74197c40 0x7f6b7419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b74103230 0x7f6b74197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 --2- 172.21.15.189:0/1194412874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b741977c0 0x7f6b7419c830 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 >> 172.21.15.189:0/1194412874 conn(0x7f6b740fd0d0 msgr2=0x7f6b740fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 shutdown_connections 2024-03-16T20:27:52.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:27:52.809+0000 7f6b78d13700 1 -- 172.21.15.189:0/1194412874 wait complete. 2024-03-16T20:27:52.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:52.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: Reconfiguring mon.b (monmap changed)... 2024-03-16T20:27:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: Reconfiguring daemon mon.b on smithi031 2024-03-16T20:27:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:52 smithi023 ceph-mon[31879]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:53.341 DEBUG:teuthology.orchestra.run.smithi031:mgr.y> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T20:27:53.344 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.z.service 2024-03-16T20:27:53.346 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T20:27:53.347 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:27:53.347 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T20:27:53.365 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T20:27:53.365 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-16 20:26:59.949466417 +0000 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-16 20:21:59.344462484 +0000 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-16 20:21:59.344462484 +0000 2024-03-16T20:27:53.422 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-16 20:21:59.344462484 +0000 2024-03-16T20:27:53.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T20:27:53.491 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-16T20:27:53.491 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-16T20:27:53.491 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000302031 s, 1.7 MB/s 2024-03-16T20:27:53.493 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T20:27:53.550 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-16 20:26:59.979466186 +0000 2024-03-16T20:27:53.607 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-16 20:21:59.691460117 +0000 2024-03-16T20:27:53.608 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-16 20:21:59.691460117 +0000 2024-03-16T20:27:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-16 20:21:59.691460117 +0000 2024-03-16T20:27:53.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T20:27:53.676 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-16T20:27:53.676 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-16T20:27:53.676 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000267784 s, 1.9 MB/s 2024-03-16T20:27:53.677 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T20:27:53.733 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-16 20:27:00.005465985 +0000 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-16 20:22:00.042457722 +0000 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-16 20:22:00.042457722 +0000 2024-03-16T20:27:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-16 20:22:00.042457722 +0000 2024-03-16T20:27:53.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T20:27:53.855 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-16T20:27:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-16T20:27:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000301369 s, 1.7 MB/s 2024-03-16T20:27:53.857 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T20:27:53.913 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-16T20:27:53.969 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T20:27:53.969 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:53.969 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-16 20:27:00.030465793 +0000 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-16 20:22:00.403455260 +0000 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-16 20:22:00.403455260 +0000 2024-03-16T20:27:53.970 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-16 20:22:00.403455260 +0000 2024-03-16T20:27:53.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi031=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: Saving service mgr spec with placement smithi023=x;smithi031=y;smithi189=z;count:3 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:53.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-16T20:27:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-16T20:27:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000293877 s, 1.7 MB/s 2024-03-16T20:27:54.037 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi031=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: Saving service mgr spec with placement smithi023=x;smithi031=y;smithi189=z;count:3 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:54.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:54.094 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-03-16T20:27:54.094 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T20:27:54.113 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T20:27:54.113 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-03-16T20:27:54.171 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T20:27:54.171 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:54.171 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-16 20:27:41.489963914 +0000 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-16 20:22:29.644145942 +0000 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-16 20:22:29.644145942 +0000 2024-03-16T20:27:54.172 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-16 20:22:29.644145942 +0000 2024-03-16T20:27:54.172 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi031=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: Saving service mgr spec with placement smithi023=x;smithi031=y;smithi189=z;count:3 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T20:27:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-16T20:27:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-16T20:27:54.220 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000267752 s, 1.9 MB/s 2024-03-16T20:27:54.221 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T20:27:54.280 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-03-16T20:27:54.337 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T20:27:54.337 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:54.337 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-16T20:27:54.337 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-16 20:27:41.490963896 +0000 2024-03-16T20:27:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-16 20:22:29.997139758 +0000 2024-03-16T20:27:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-16 20:22:29.997139758 +0000 2024-03-16T20:27:54.338 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-16 20:22:29.997139758 +0000 2024-03-16T20:27:54.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T20:27:54.409 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-16T20:27:54.409 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-16T20:27:54.409 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00021342 s, 2.4 MB/s 2024-03-16T20:27:54.410 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T20:27:54.469 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-16 20:27:41.490963896 +0000 2024-03-16T20:27:54.526 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-16 20:22:30.343133697 +0000 2024-03-16T20:27:54.527 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-16 20:22:30.343133697 +0000 2024-03-16T20:27:54.527 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-16 20:22:30.343133697 +0000 2024-03-16T20:27:54.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T20:27:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-16T20:27:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-16T20:27:54.598 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000195286 s, 2.6 MB/s 2024-03-16T20:27:54.599 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T20:27:54.656 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-03-16 20:27:41.491963878 +0000 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-03-16 20:22:30.695127531 +0000 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-03-16 20:22:30.695127531 +0000 2024-03-16T20:27:54.712 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-03-16 20:22:30.695127531 +0000 2024-03-16T20:27:54.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T20:27:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-03-16T20:27:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-03-16T20:27:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000236505 s, 2.2 MB/s 2024-03-16T20:27:54.784 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T20:27:54.841 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-16T20:27:54.841 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T20:27:54.858 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T20:27:54.858 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-16T20:27:54.913 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T20:27:54.913 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-16 20:27:32.007708348 +0000 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-16 20:22:24.364482567 +0000 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-16 20:22:24.364482567 +0000 2024-03-16T20:27:54.914 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-16 20:22:24.364482567 +0000 2024-03-16T20:27:54.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T20:27:54.936 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: Reconfiguring mon.c (monmap changed)... 2024-03-16T20:27:54.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: Reconfiguring daemon mon.c on smithi189 2024-03-16T20:27:54.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:54.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:54.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:54 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-16T20:27:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-16T20:27:54.981 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000295288 s, 1.7 MB/s 2024-03-16T20:27:54.982 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T20:27:55.039 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: Reconfiguring mon.c (monmap changed)... 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: Reconfiguring daemon mon.c on smithi189 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:55.093 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T20:27:55.093 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-16 20:27:32.008708328 +0000 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-16 20:22:24.694476208 +0000 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-16 20:22:24.694476208 +0000 2024-03-16T20:27:55.094 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-16 20:22:24.694476208 +0000 2024-03-16T20:27:55.094 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: Reconfiguring mon.c (monmap changed)... 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: Reconfiguring daemon mon.c on smithi189 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:55.158 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:54 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:27:55.160 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-16T20:27:55.160 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-16T20:27:55.160 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000229071 s, 2.2 MB/s 2024-03-16T20:27:55.161 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T20:27:55.216 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-16 20:27:32.008708328 +0000 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-16 20:22:25.013470060 +0000 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-16 20:22:25.013470060 +0000 2024-03-16T20:27:55.271 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-16 20:22:25.013470060 +0000 2024-03-16T20:27:55.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T20:27:55.336 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-16T20:27:55.336 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-16T20:27:55.336 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000199956 s, 2.6 MB/s 2024-03-16T20:27:55.337 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T20:27:55.393 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-16T20:27:55.448 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-16 20:27:32.008708328 +0000 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-16 20:22:25.377463046 +0000 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-16 20:22:25.377463046 +0000 2024-03-16T20:27:55.449 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-16 20:22:25.377463046 +0000 2024-03-16T20:27:55.449 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T20:27:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-16T20:27:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-16T20:27:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000240585 s, 2.1 MB/s 2024-03-16T20:27:55.516 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T20:27:55.572 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-16T20:27:55.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T20:27:55.806 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:55.928 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:56.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:56.938 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:56 smithi189 ceph-mon[36655]: Deploying daemon mgr.z on smithi189 2024-03-16T20:27:56.938 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:56 smithi189 ceph-mon[36655]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:57.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:56 smithi023 ceph-mon[31879]: Deploying daemon mgr.z on smithi189 2024-03-16T20:27:57.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:56 smithi023 ceph-mon[31879]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:57.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:56 smithi031 ceph-mon[36281]: Deploying daemon mgr.z on smithi189 2024-03-16T20:27:57.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:56 smithi031 ceph-mon[36281]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:57.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 podman[38493]: 2024-03-16 20:27:57.054031248 +0000 UTC m=+0.017427983 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:27:57.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 podman[38493]: 2024-03-16 20:27:57.357165809 +0000 UTC m=+0.320562555 container init 89a5d412cebec62b1eb016f17e8ca6a6be23b7128bcc4659bb3b76ce1b0028bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:27:57.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 podman[38493]: 2024-03-16 20:27:57.360009697 +0000 UTC m=+0.323406426 container start 89a5d412cebec62b1eb016f17e8ca6a6be23b7128bcc4659bb3b76ce1b0028bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T20:27:57.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 ceph-mgr[38512]: -- 172.21.15.189:0/3205267104 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639727da1a0 con 0x5639727b1000 2024-03-16T20:27:57.720 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 bash[38493]: 89a5d412cebec62b1eb016f17e8ca6a6be23b7128bcc4659bb3b76ce1b0028bd 2024-03-16T20:27:57.720 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 systemd[1]: Started Ceph mgr.z for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:27:57.801 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T20:27:57.801 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T20:27:57.801 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-16T20:27:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-16T20:27:57.802 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00706884 s, 1.5 GB/s 2024-03-16T20:27:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-16T20:27:58.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:57.899+0000 7fbaeb0c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T20:27:58.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:57 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:57.992+0000 7fbaeb0c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T20:27:58.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-16T20:27:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:27:58.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: Deploying daemon mgr.y on smithi031 2024-03-16T20:27:58.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:58 smithi023 ceph-mon[31879]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:58.631+0000 7fbaeb0c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:58.723+0000 7fbaeb0c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:58.863+0000 7fbaeb0c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:58.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:58.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:58.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:58.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: Deploying daemon mgr.y on smithi031 2024-03-16T20:27:58.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-mon[36655]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: Deploying daemon mgr.y on smithi031 2024-03-16T20:27:58.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:58 smithi031 ceph-mon[36281]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:27:59.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:58.959+0000 7fbaeb0c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T20:27:59.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.042+0000 7fbaeb0c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T20:27:59.489 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.308+0000 7fbaeb0c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T20:27:59.489 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.422+0000 7fbaeb0c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T20:27:59.804 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.513+0000 7fbaeb0c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T20:27:59.804 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.614+0000 7fbaeb0c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T20:27:59.804 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:27:59.800+0000 7fbaeb0c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T20:27:59.977 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:27:59 smithi031 podman[37643]: 2024-03-16 20:27:59.809336544 +0000 UTC m=+0.018674320 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:28:00.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:27:59 smithi023 ceph-mon[31879]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- 172.21.15.23:0/1467405245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80100f20 msgr2=0x7fae80101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 --2- 172.21.15.23:0/1467405245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80100f20 0x7fae80101340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009990 tx=0x7fae7c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- 172.21.15.23:0/1467405245 shutdown_connections 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 --2- 172.21.15.23:0/1467405245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80100f20 0x7fae80101340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 --2- 172.21.15.23:0/1467405245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102a40 0x7fae80109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 --2- 172.21.15.23:0/1467405245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80102080 0x7fae80102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- 172.21.15.23:0/1467405245 >> 172.21.15.23:0/1467405245 conn(0x7fae800fc4e0 msgr2=0x7fae800fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- 172.21.15.23:0/1467405245 shutdown_connections 2024-03-16T20:28:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- 172.21.15.23:0/1467405245 wait complete. 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 Processor -- start 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.071+0000 7fae880c8700 1 -- start start 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80100f20 0x7fae80196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102080 0x7fae80197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019ea70 con 0x7fae80102a40 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019ebb0 con 0x7fae80102080 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8006a7c0 con 0x7fae80100f20 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55626/0 (socket says 172.21.15.23:55626) 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 -- 172.21.15.23:0/297337947 learned_addr learned my addr 172.21.15.23:0/297337947 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:28:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae85663700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102080 0x7fae80197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae85e64700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80100f20 0x7fae80196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 -- 172.21.15.23:0/297337947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80100f20 msgr2=0x7fae80196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80100f20 0x7fae80196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 -- 172.21.15.23:0/297337947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102080 msgr2=0x7fae80197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102080 0x7fae80197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 -- 172.21.15.23:0/297337947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0095f0 con 0x7fae80102a40 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae86665700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae7800eb80 tx=0x7fae7800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78018070 con 0x7fae80102a40 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae7800f040 con 0x7fae80102a40 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78004020 con 0x7fae80102a40 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8006aa40 con 0x7fae80102a40 2024-03-16T20:28:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.072+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8006af30 con 0x7fae80102a40 2024-03-16T20:28:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.073+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae801082d0 con 0x7fae80102a40 2024-03-16T20:28:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.073+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fae78003a50 con 0x7fae80102a40 2024-03-16T20:28:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.073+0000 7fae76ffd700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 0x7fae6c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.074+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae7804b480 con 0x7fae80102a40 2024-03-16T20:28:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.074+0000 7fae85e64700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 0x7fae6c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.074+0000 7fae85e64700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 0x7fae6c03b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00b7c0 tx=0x7fae7c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:00.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.078+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fae780137c0 con 0x7fae80102a40 2024-03-16T20:28:00.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:27:59 smithi189 ceph-mon[36655]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:00.229 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:27:59 smithi031 ceph-mon[36281]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:00.229 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 podman[37643]: 2024-03-16 20:28:00.14052262 +0000 UTC m=+0.349860403 container init a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T20:28:00.229 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 podman[37643]: 2024-03-16 20:28:00.144288273 +0000 UTC m=+0.353626030 container start a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-16T20:28:00.230 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mgr[37663]: -- 172.21.15.31:0/738948880 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2fdd4e1a0 con 0x55b2fdd25000 2024-03-16T20:28:00.230 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 bash[37643]: a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 2024-03-16T20:28:00.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:00.291+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fae8002da30 con 0x7fae6c038f90 2024-03-16T20:28:00.687 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 systemd[1]: Started Ceph mgr.y for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:00.743+0000 7f4c83977200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:00.843+0000 7f4c83977200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:01.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:01.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:01.221 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:00.814+0000 7fbaeb0c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T20:28:01.222 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:00.949+0000 7fbaeb0c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T20:28:01.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:01.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:01.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:01.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:01.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:01.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:01.230 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:01.623 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:01 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:01.325+0000 7fbaeb0c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T20:28:01.623 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:01 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:01.417+0000 7fbaeb0c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T20:28:01.623 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:01 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:01.619+0000 7fbaeb0c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:28:01.653 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:01 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:01.554+0000 7f4c83977200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T20:28:01.653 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:01 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:01.649+0000 7f4c83977200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T20:28:01.904 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:01 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:01.798+0000 7f4c83977200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T20:28:01.904 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:01 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:01.901+0000 7f4c83977200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T20:28:01.969 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:01 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:01.810+0000 7fbaeb0c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T20:28:02.186 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:01 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:01.987+0000 7f4c83977200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T20:28:02.364 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:02.103+0000 7fbaeb0c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:28:02.559 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:02.258+0000 7f4c83977200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T20:28:02.560 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:02.371+0000 7f4c83977200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T20:28:02.560 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:02.459+0000 7f4c83977200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T20:28:02.560 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:02.556+0000 7f4c83977200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T20:28:02.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:02.360+0000 7fbaeb0c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T20:28:02.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:02.595+0000 7fbaeb0c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T20:28:02.863 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:02.736+0000 7f4c83977200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]: dispatch 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]': finished 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T20:28:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:02 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]: dispatch 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]': finished 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T20:28:03.215 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:02 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:03.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:03.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]: dispatch 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1001388398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96865589-182e-46bf-b061-3b6f0521156b"}]': finished 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T20:28:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:02 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:03.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:03 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:03.399+0000 7fbaeb0c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T20:28:04.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:03 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:03.753+0000 7f4c83977200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T20:28:04.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:03 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:03.886+0000 7f4c83977200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T20:28:04.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:03 smithi031 ceph-mon[36281]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T20:28:04.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:03 smithi031 ceph-mon[36281]: Reconfiguring daemon mgr.x on smithi023 2024-03-16T20:28:04.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:03 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3459997528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:04.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:03 smithi189 ceph-mon[36655]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T20:28:04.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:03 smithi189 ceph-mon[36655]: Reconfiguring daemon mgr.x on smithi023 2024-03-16T20:28:04.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:03 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3459997528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:04.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:03 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:03.974+0000 7fbaeb0c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T20:28:04.219 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:04.072+0000 7fbaeb0c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T20:28:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:03 smithi023 ceph-mon[31879]: Reconfiguring mgr.x (unknown last config time)... 2024-03-16T20:28:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:03 smithi023 ceph-mon[31879]: Reconfiguring daemon mgr.x on smithi023 2024-03-16T20:28:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3459997528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:04.523 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:04.230+0000 7f4c83977200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T20:28:04.524 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:04.321+0000 7f4c83977200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T20:28:04.524 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:04.520+0000 7f4c83977200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:28:04.719 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z[38508]: 2024-03-16T20:28:04.287+0000 7fbaeb0c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T20:28:04.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:04.891+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 14) v1 ==== 91024+0+0 (secure 0 0 0) 0x7fae78019d30 con 0x7fae80102a40 2024-03-16T20:28:04.902 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:04.708+0000 7f4c83977200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:05.000+0000 7f4c83977200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: Standby manager daemon z started 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T20:28:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:04 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:05.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:05.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: Standby manager daemon z started 2024-03-16T20:28:05.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T20:28:05.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:05.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T20:28:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:04 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:05.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:05.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: Standby manager daemon z started 2024-03-16T20:28:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-16T20:28:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-16T20:28:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:04 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.189:0/1132588322' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:05.687 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:05.266+0000 7f4c83977200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T20:28:05.687 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:05.507+0000 7f4c83977200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T20:28:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-mon[36281]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-16T20:28:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T20:28:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:06.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:05 smithi189 ceph-mon[36655]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-16T20:28:06.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T20:28:06.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:06.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:05 smithi023 ceph-mon[31879]: mgrmap e14: x(active, since 75s), standbys: z 2024-03-16T20:28:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-16T20:28:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:06.686 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:06 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:06.333+0000 7f4c83977200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T20:28:07.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:06 smithi023 ceph-mon[31879]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:07.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:06 smithi031 ceph-mon[36281]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:07.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:06 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:06.918+0000 7f4c83977200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T20:28:07.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:07.017+0000 7f4c83977200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T20:28:07.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:06 smithi189 ceph-mon[36655]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:07.687 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y[37658]: 2024-03-16T20:28:07.236+0000 7f4c83977200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T20:28:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:07.918+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fae78004180 con 0x7fae80102a40 2024-03-16T20:28:08.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:07 smithi023 ceph-mon[31879]: Standby manager daemon y started 2024-03-16T20:28:08.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:07 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T20:28:08.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:07 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:08.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:07 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T20:28:08.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:07 smithi023 ceph-mon[31879]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:08.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-mon[36281]: Standby manager daemon y started 2024-03-16T20:28:08.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T20:28:08.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:08.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T20:28:08.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:07 smithi031 ceph-mon[36281]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:08.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:07 smithi189 ceph-mon[36655]: Standby manager daemon y started 2024-03-16T20:28:08.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:07 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-16T20:28:08.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:07 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T20:28:08.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:07 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-16T20:28:08.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:07 smithi189 ceph-mon[36655]: from='mgr.? 172.21.15.31:0/2103641556' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:09.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:09.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-16T20:28:09.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:11.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:10 smithi023 ceph-mon[31879]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:11.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T20:28:11.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:11.186 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:10 smithi031 ceph-mon[36281]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:11.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T20:28:11.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:11.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:10 smithi189 ceph-mon[36655]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:11.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T20:28:11.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:12.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:11 smithi023 ceph-mon[31879]: Deploying daemon osd.0 on smithi023 2024-03-16T20:28:12.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:11 smithi031 ceph-mon[36281]: Deploying daemon osd.0 on smithi023 2024-03-16T20:28:12.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:11 smithi189 ceph-mon[36655]: Deploying daemon osd.0 on smithi023 2024-03-16T20:28:13.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:12 smithi031 ceph-mon[36281]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:13.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:12 smithi189 ceph-mon[36655]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:13.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:12 smithi023 ceph-mon[31879]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:14.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:13 smithi031 ceph-mon[36281]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:14.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:13 smithi189 ceph-mon[36655]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:14.325 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:13 smithi023 ceph-mon[31879]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:16.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:16.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:16 smithi023 ceph-mon[31879]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:16.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:16.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:16 smithi031 ceph-mon[36281]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:16.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:16.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:16.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:16 smithi189 ceph-mon[36655]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:18.660 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-16T20:28:18.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.656+0000 7fae76ffd700 1 -- 172.21.15.23:0/297337947 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fae8002da30 con 0x7fae6c038f90 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:18 smithi023 ceph-mon[31879]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:18.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 msgr2=0x7fae6c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:18.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 0x7fae6c03b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00b7c0 tx=0x7fae7c005fb0 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 msgr2=0x7fae8019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae7800eb80 tx=0x7fae7800ee90 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 shutdown_connections 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fae6c038f90 0x7fae6c03b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80102a40 0x7fae8019e4a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae80100f20 0x7fae80196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 --2- 172.21.15.23:0/297337947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae80102080 0x7fae80197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 >> 172.21.15.23:0/297337947 conn(0x7fae800fc4e0 msgr2=0x7fae80105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 shutdown_connections 2024-03-16T20:28:18.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:28:18.659+0000 7fae880c8700 1 -- 172.21.15.23:0/297337947 wait complete. 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:18 smithi031 ceph-mon[36281]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:18.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:18.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:18.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:18.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:18 smithi189 ceph-mon[36655]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:19.249 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T20:28:19.251 INFO:tasks.cephadm:Deploying osd.1 on smithi031 with /dev/vg_nvme/lv_4... 2024-03-16T20:28:19.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T20:28:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:28:19.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:19.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:19 smithi023 ceph-mon[31879]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T20:28:19.898 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:19.898 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.898 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.898 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:19 smithi031 ceph-mon[36281]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T20:28:19.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:19.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:19.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:19 smithi189 ceph-mon[36655]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T20:28:20.759 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:20 smithi031 ceph-mon[36281]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T20:28:20.759 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:20 smithi031 ceph-mon[36281]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T20:28:20.759 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:20 smithi031 ceph-mon[36281]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:28:20.759 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:20.759 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:20 smithi031 ceph-mon[36281]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:20.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:20 smithi023 ceph-mon[31879]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T20:28:20.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:20 smithi023 ceph-mon[31879]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T20:28:20.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:20 smithi023 ceph-mon[31879]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:28:20.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:20.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:20 smithi023 ceph-mon[31879]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:20.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:20 smithi189 ceph-mon[36655]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T20:28:20.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:20 smithi189 ceph-mon[36655]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T20:28:20.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:20 smithi189 ceph-mon[36655]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:28:20.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:20.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:20 smithi189 ceph-mon[36655]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111478 s, 941 MB/s 2024-03-16T20:28:21.320 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-16T20:28:21.339 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0[43983]: 2024-03-16T20:28:21.046+0000 7f88e097a700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T20:28:21.339 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0[43983]: 2024-03-16T20:28:21.049+0000 7f88d8765700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:28:21.875 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:21 smithi031 ceph-mon[36281]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:28:21.875 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:21 smithi031 ceph-mon[36281]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T20:28:21.875 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.875 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.875 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:21 smithi031 ceph-mon[36281]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' 2024-03-16T20:28:21.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-03-16T20:28:21.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-mon[31879]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:28:21.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-mon[31879]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T20:28:21.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.945 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:21 smithi023 ceph-mon[31879]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' 2024-03-16T20:28:21.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:21 smithi189 ceph-mon[36655]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:28:21.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:21 smithi189 ceph-mon[36655]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T20:28:21.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:21.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:21 smithi189 ceph-mon[36655]: from='osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785]' entity='osd.0' 2024-03-16T20:28:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:28:22.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:28:22.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:28:22.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:22.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] boot 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T20:28:22.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:22.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:28:22.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] boot 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T20:28:22.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:23.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:23.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: osd.0 [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] boot 2024-03-16T20:28:23.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T20:28:23.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T20:28:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 -- 172.21.15.31:0/2798970887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894103bb0 msgr2=0x7fa89410a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 --2- 172.21.15.31:0/2798970887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894103bb0 0x7fa89410a460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00b0d0 tx=0x7fa88c00b3e0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 -- 172.21.15.31:0/2798970887 shutdown_connections 2024-03-16T20:28:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 --2- 172.21.15.31:0/2798970887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894102010 0x7fa894102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 --2- 172.21.15.31:0/2798970887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894103bb0 0x7fa89410a460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 --2- 172.21.15.31:0/2798970887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa894103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 -- 172.21.15.31:0/2798970887 >> 172.21.15.31:0/2798970887 conn(0x7fa8940fd5f0 msgr2=0x7fa8940ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 -- 172.21.15.31:0/2798970887 shutdown_connections 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.556+0000 7fa899389700 1 -- 172.21.15.31:0/2798970887 wait complete. 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 Processor -- start 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 -- start start 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa8941985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 0x7fa89419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89419fc00 con 0x7fa894103bb0 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89419fd40 con 0x7fa894102010 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa899389700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89419fed0 con 0x7fa894103210 2024-03-16T20:28:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa892ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa892ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40690/0 (socket says 172.21.15.31:40690) 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa892ffd700 1 -- 172.21.15.31:0/1000248839 learned_addr learned my addr 172.21.15.31:0/1000248839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.557+0000 7fa8927fc700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa8941985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa8937fe700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 0x7fa89419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 -- 172.21.15.31:0/1000248839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 msgr2=0x7fa8941985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa8941985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 -- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 msgr2=0x7fa89419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 0x7fa89419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 -- 172.21.15.31:0/1000248839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c009d00 con 0x7fa894102010 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa8927fc700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa8941985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa8937fe700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 0x7fa89419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa892ffd700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c970 tx=0x7fa88400cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884004220 con 0x7fa894102010 2024-03-16T20:28:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8941a01b0 con 0x7fa894102010 2024-03-16T20:28:23.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa884007d80 con 0x7fa894102010 2024-03-16T20:28:23.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884003da0 con 0x7fa894102010 2024-03-16T20:28:23.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.558+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8941a06b0 con 0x7fa894102010 2024-03-16T20:28:23.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.559+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8941093d0 con 0x7fa894102010 2024-03-16T20:28:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.560+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fa8840078a0 con 0x7fa894102010 2024-03-16T20:28:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.561+0000 7fa87bfff700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 0x7fa87c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.561+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fa8840cb980 con 0x7fa894102010 2024-03-16T20:28:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.561+0000 7fa8927fc700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 0x7fa87c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:23.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.561+0000 7fa8927fc700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 0x7fa87c0a3ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa888005950 tx=0x7fa8880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:23.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.565+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa884096600 con 0x7fa894102010 2024-03-16T20:28:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:23.777+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa89402db10 con 0x7fa87c0a1a30 2024-03-16T20:28:23.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:23 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi023 2024-03-16T20:28:23.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:23 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-16T20:28:23.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:23 smithi031 ceph-mon[36281]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T20:28:23.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:23 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi023 2024-03-16T20:28:23.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:23 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-16T20:28:23.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:23 smithi189 ceph-mon[36655]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T20:28:24.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:23 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi023 2024-03-16T20:28:24.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:23 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-16T20:28:24.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:23 smithi023 ceph-mon[31879]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T20:28:24.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:24 smithi031 ceph-mon[36281]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:24.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:24.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:24.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:24.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:24 smithi031 ceph-mon[36281]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:24.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:24 smithi189 ceph-mon[36655]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:24.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:24.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:24.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:24.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:24 smithi189 ceph-mon[36655]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:25.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:24 smithi023 ceph-mon[31879]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:25.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:25.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:25.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:25.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:24 smithi023 ceph-mon[31879]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:26.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:25 smithi023 ceph-mon[31879]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:26.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:25 smithi031 ceph-mon[36281]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:26.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:25 smithi189 ceph-mon[36655]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:27.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/4076143271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]': finished 2024-03-16T20:28:27.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T20:28:27.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:27.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:26 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/4140079022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/4076143271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]': finished 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:27.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:26 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/4140079022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:27.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/4076143271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]: dispatch 2024-03-16T20:28:27.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3"}]': finished 2024-03-16T20:28:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T20:28:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:26 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/4140079022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:28.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:27 smithi023 ceph-mon[31879]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:28.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:27 smithi031 ceph-mon[36281]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:28.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:27 smithi189 ceph-mon[36655]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:30.238 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:29 smithi031 ceph-mon[36281]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:30.271 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:29 smithi189 ceph-mon[36655]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:30.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:29 smithi023 ceph-mon[31879]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:32.163 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:31 smithi031 ceph-mon[36281]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:32.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:31 smithi023 ceph-mon[31879]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:32.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:31 smithi189 ceph-mon[36655]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:33.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:33 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T20:28:33.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:33 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:33 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:33 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:33.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T20:28:33.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:34.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:34 smithi031 ceph-mon[36281]: Deploying daemon osd.1 on smithi031 2024-03-16T20:28:34.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:34 smithi031 ceph-mon[36281]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:34.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:34 smithi023 ceph-mon[31879]: Deploying daemon osd.1 on smithi031 2024-03-16T20:28:34.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:34 smithi023 ceph-mon[31879]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:34.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:34 smithi189 ceph-mon[36655]: Deploying daemon osd.1 on smithi031 2024-03-16T20:28:34.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:34 smithi189 ceph-mon[36655]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:36.074 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:35 smithi031 ceph-mon[36281]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:36.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:35 smithi023 ceph-mon[31879]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:36.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:35 smithi189 ceph-mon[36655]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:38.133 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:37 smithi031 ceph-mon[36281]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:38.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:37 smithi023 ceph-mon[31879]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:38.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:37 smithi189 ceph-mon[36655]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:40.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:40.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:40.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:40.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:41.106 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:40 smithi031 ceph-mon[36281]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:41.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:40 smithi189 ceph-mon[36655]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:41.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:40 smithi023 ceph-mon[31879]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:42.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.098+0000 7fa87bfff700 1 -- 172.21.15.31:0/1000248839 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa89402db10 con 0x7fa87c0a1a30 2024-03-16T20:28:42.102 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 1 on host 'smithi031' 2024-03-16T20:28:42.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 msgr2=0x7fa87c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:42.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 0x7fa87c0a3ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa888005950 tx=0x7fa8880058e0 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 msgr2=0x7fa894198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa88400c970 tx=0x7fa88400cd30 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 shutdown_connections 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa87c0a1a30 0x7fa87c0a3ef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa894103bb0 0x7fa89419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa894102010 0x7fa894198070 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 --2- 172.21.15.31:0/1000248839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa894103210 0x7fa8941985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 >> 172.21.15.31:0/1000248839 conn(0x7fa8940fd5f0 msgr2=0x7fa894106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 shutdown_connections 2024-03-16T20:28:42.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:28:42.101+0000 7fa899389700 1 -- 172.21.15.31:0/1000248839 wait complete. 2024-03-16T20:28:42.668 DEBUG:teuthology.orchestra.run.smithi031:osd.1> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T20:28:42.670 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/vg_nvme/lv_4... 2024-03-16T20:28:42.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-16T20:28:42.916 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:28:42.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:42.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:42.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:42.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:42.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:42.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:42.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:42 smithi189 ceph-mon[36655]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:43.152 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-mon[36281]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:43.153 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:28:42 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:28:42.838+0000 7f382d7e17c0 -1 osd.1 0 log_to_monitors true 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:42 smithi023 ceph-mon[31879]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T20:28:44.406 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T20:28:44.406 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.406 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T20:28:44.406 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:44.406 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.407 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:44 smithi189 ceph-mon[36655]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-mon[36281]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: from='osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:28:44.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:44 smithi023 ceph-mon[31879]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:44.937 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:28:44.620+0000 7f3822e46700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T20:28:44.937 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:28:44 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:28:44.624+0000 7f381ac31700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111845 s, 938 MB/s 2024-03-16T20:28:44.975 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-16T20:28:45.276 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:28:45.276 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T20:28:45.277 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.277 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.277 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: from='osd.1 ' entity='osd.1' 2024-03-16T20:28:45.277 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: from='osd.1 ' entity='osd.1' 2024-03-16T20:28:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.571 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-03-16T20:28:45.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:28:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T20:28:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: from='osd.1 ' entity='osd.1' 2024-03-16T20:28:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:45.816 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi031 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi031 to 17084M 2024-03-16T20:28:46.813 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] boot 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.814 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:46 smithi189 ceph-mon[36655]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi031 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi031 to 17084M 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] boot 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:46 smithi023 ceph-mon[31879]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi031 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi031 to 17084M 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: osd.1 [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] boot 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:28:46.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:46 smithi031 ceph-mon[36281]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 -- 172.21.15.189:0/1338673083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 msgr2=0x7ff7081025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 --2- 172.21.15.189:0/1338673083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081025e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009990 tx=0x7ff6fc009ca0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 -- 172.21.15.189:0/1338673083 shutdown_connections 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 --2- 172.21.15.189:0/1338673083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708100f80 0x7ff7081013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 --2- 172.21.15.189:0/1338673083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708102b20 0x7ff7081093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 --2- 172.21.15.189:0/1338673083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081025e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 -- 172.21.15.189:0/1338673083 >> 172.21.15.189:0/1338673083 conn(0x7ff7080fc540 msgr2=0x7ff7080fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:28:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 -- 172.21.15.189:0/1338673083 shutdown_connections 2024-03-16T20:28:47.311 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.306+0000 7ff70d751700 1 -- 172.21.15.189:0/1338673083 wait complete. 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.307+0000 7ff70d751700 1 Processor -- start 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.307+0000 7ff70d751700 1 -- start start 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 0x7ff708196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708102b20 0x7ff70819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70819ea30 con 0x7ff708102b20 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70819eb70 con 0x7ff708100f80 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70819ed00 con 0x7ff708102180 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:47.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:34862/0 (socket says 172.21.15.189:34862) 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 -- 172.21.15.189:0/1943035058 learned_addr learned my addr 172.21.15.189:0/1943035058 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7077fe700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708102b20 0x7ff70819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff706ffd700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 0x7ff708196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 -- 172.21.15.189:0/1943035058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 msgr2=0x7ff708196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 0x7ff708196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 -- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708102b20 msgr2=0x7ff70819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708102b20 0x7ff70819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 -- 172.21.15.189:0/1943035058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc0095f0 con 0x7ff708102180 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff706ffd700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 0x7ff708196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff7067fc700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc000c00 tx=0x7ff6fc00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc01d070 con 0x7ff708102180 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6fc00fbf0 con 0x7ff708102180 2024-03-16T20:28:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70819ef80 con 0x7ff708102180 2024-03-16T20:28:47.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0173f0 con 0x7ff708102180 2024-03-16T20:28:47.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.308+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70819f330 con 0x7ff708102180 2024-03-16T20:28:47.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.310+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff708108410 con 0x7ff708102180 2024-03-16T20:28:47.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.310+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7ff6fc017550 con 0x7ff708102180 2024-03-16T20:28:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.311+0000 7ff6effff700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 0x7ff6f00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:28:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.311+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7ff6fc0cce00 con 0x7ff708102180 2024-03-16T20:28:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.311+0000 7ff706ffd700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 0x7ff6f00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:28:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.312+0000 7ff706ffd700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 0x7ff6f00a3ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8005fd0 tx=0x7ff6f8005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:28:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.316+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff6fc0978f0 con 0x7ff708102180 2024-03-16T20:28:47.522 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:28:47.519+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff70802da90 con 0x7ff6f00a1a30 2024-03-16T20:28:48.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:47 smithi023 ceph-mon[31879]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T20:28:48.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:48.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:48.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:48.125 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:47 smithi189 ceph-mon[36655]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T20:28:48.126 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:48.126 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:48.126 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:47 smithi031 ceph-mon[36281]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T20:28:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:28:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:28:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:49.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:48 smithi023 ceph-mon[31879]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:49.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:48 smithi023 ceph-mon[31879]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:49.144 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:48 smithi189 ceph-mon[36655]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:49.144 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:48 smithi189 ceph-mon[36655]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:49.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:48 smithi031 ceph-mon[36281]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:28:49.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:48 smithi031 ceph-mon[36281]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:49 smithi023 ceph-mon[31879]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:50.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:49 smithi189 ceph-mon[36655]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:50.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:49 smithi031 ceph-mon[36281]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/3847483547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]': finished 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:28:51.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:50 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/2305154015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/3847483547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]': finished 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:28:51.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:50 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/2305154015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:51.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/3847483547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]: dispatch 2024-03-16T20:28:51.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07b0e4ef-221b-4cfa-b176-8ff89dda1c90"}]': finished 2024-03-16T20:28:51.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T20:28:51.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:28:51.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:50 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/2305154015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:28:52.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:51 smithi023 ceph-mon[31879]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:52.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:51 smithi031 ceph-mon[36281]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:52.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:51 smithi189 ceph-mon[36655]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:54.330 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:53 smithi189 ceph-mon[36655]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:54.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:53 smithi023 ceph-mon[31879]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:54.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:53 smithi031 ceph-mon[36281]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:56.056 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:55 smithi189 ceph-mon[36655]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:56.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:55 smithi023 ceph-mon[31879]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:56.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:55 smithi031 ceph-mon[36281]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:57.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:57 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T20:28:57.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:57 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:57.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:57 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T20:28:57.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:57 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:57.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:57 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T20:28:57.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:57 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:28:58.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:58 smithi189 ceph-mon[36655]: Deploying daemon osd.2 on smithi189 2024-03-16T20:28:58.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:58 smithi189 ceph-mon[36655]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:58.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:58 smithi031 ceph-mon[36281]: Deploying daemon osd.2 on smithi189 2024-03-16T20:28:58.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:58 smithi031 ceph-mon[36281]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:28:58.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:58 smithi023 ceph-mon[31879]: Deploying daemon osd.2 on smithi189 2024-03-16T20:28:58.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:58 smithi023 ceph-mon[31879]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:00.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:28:59 smithi189 ceph-mon[36655]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:00.239 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:28:59 smithi031 ceph-mon[36281]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:28:59 smithi023 ceph-mon[31879]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:02.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:01 smithi189 ceph-mon[36655]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:01 smithi023 ceph-mon[31879]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:02.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:01 smithi031 ceph-mon[36281]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:03.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:03.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:03.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:03.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:04.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:04 smithi031 ceph-mon[36281]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:04.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:04 smithi189 ceph-mon[36655]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:04.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:04 smithi023 ceph-mon[31879]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:06.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:06.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:06.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:06.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:06.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.086+0000 7ff6effff700 1 -- 172.21.15.189:0/1943035058 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff70802da90 con 0x7ff6f00a1a30 2024-03-16T20:29:06.091 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-03-16T20:29:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 msgr2=0x7ff6f00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 0x7ff6f00a3ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8005fd0 tx=0x7ff6f8005f00 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 msgr2=0x7ff7081973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc000c00 tx=0x7ff6fc00f740 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 shutdown_connections 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff6f00a1a30 0x7ff6f00a3ef0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff708102b20 0x7ff70819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100f80 0x7ff708196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 --2- 172.21.15.189:0/1943035058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff708102180 0x7ff7081973e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 >> 172.21.15.189:0/1943035058 conn(0x7ff7080fc540 msgr2=0x7ff708105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 shutdown_connections 2024-03-16T20:29:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:29:06.089+0000 7ff70d751700 1 -- 172.21.15.189:0/1943035058 wait complete. 2024-03-16T20:29:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:06.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:06.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.729 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T20:29:06.732 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-16T20:29:06.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T20:29:06.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:06.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:06.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:06.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:06.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:06 smithi189 ceph-mon[36655]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:06.976 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:29:07.075 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:07.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:07.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:07.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:07.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:07.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:06 smithi023 ceph-mon[31879]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:07.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:07.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:07.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:07.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:07.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:07.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:06 smithi031 ceph-mon[36281]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T20:29:08.191 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T20:29:08.191 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T20:29:08.191 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:08.191 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.191 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.192 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-mon[36655]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:08 smithi023 ceph-mon[31879]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: from='osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:29:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:08 smithi031 ceph-mon[36281]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126828 s, 827 MB/s 2024-03-16T20:29:08.870 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-16T20:29:08.970 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2[41781]: 2024-03-16T20:29:08.536+0000 7efda4076700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T20:29:08.970 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:08 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2[41781]: 2024-03-16T20:29:08.542+0000 7efd9be61700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:29:09.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:09 smithi023 ceph-mon[31879]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:29:09.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:09 smithi023 ceph-mon[31879]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T20:29:09.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:09 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:09 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:09 smithi023 ceph-mon[31879]: from='osd.2 ' entity='osd.2' 2024-03-16T20:29:09.350 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:09 smithi189 ceph-mon[36655]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:29:09.350 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:09 smithi189 ceph-mon[36655]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T20:29:09.350 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:09 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.350 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:09 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.351 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:09 smithi189 ceph-mon[36655]: from='osd.2 ' entity='osd.2' 2024-03-16T20:29:09.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:09 smithi031 ceph-mon[36281]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:29:09.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:09 smithi031 ceph-mon[36281]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T20:29:09.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:09 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:09 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:09.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:09 smithi031 ceph-mon[36281]: from='osd.2 ' entity='osd.2' 2024-03-16T20:29:09.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-16T20:29:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:29:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:29:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:29:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] boot 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi189 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:10.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:10.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:10 smithi189 ceph-mon[36655]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:29:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:29:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] boot 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi189 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:10 smithi023 ceph-mon[31879]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:10.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: osd.2 [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] boot 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi189 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:10.691 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:10.692 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:10.692 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:10 smithi031 ceph-mon[36281]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-16T20:29:11.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- 172.21.15.23:0/3939396781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78102a30 msgr2=0x7f0c78109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:11.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/3939396781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78102a30 0x7f0c78109350 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000b0a0 tx=0x7f0c7000b3b0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- 172.21.15.23:0/3939396781 shutdown_connections 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/3939396781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78102a30 0x7f0c78109350 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/3939396781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78100f10 0x7f0c78101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/3939396781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102070 0x7f0c781024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- 172.21.15.23:0/3939396781 >> 172.21.15.23:0/3939396781 conn(0x7f0c780fc4b0 msgr2=0x7f0c780fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- 172.21.15.23:0/3939396781 shutdown_connections 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- 172.21.15.23:0/3939396781 wait complete. 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 Processor -- start 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- start start 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78102070 0x7f0c7819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102a30 0x7f0c781a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a1d80 con 0x7f0c78100f10 2024-03-16T20:29:11.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a1ec0 con 0x7f0c78102a30 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.332+0000 7f0c7eb22700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781a2050 con 0x7f0c78102070 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42502/0 (socket says 172.21.15.23:42502) 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 -- 172.21.15.23:0/4164315390 learned_addr learned my addr 172.21.15.23:0/4164315390 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c77fff700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78102070 0x7f0c7819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7d0bf700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102a30 0x7f0c781a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 -- 172.21.15.23:0/4164315390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78102070 msgr2=0x7f0c7819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78102070 0x7f0c7819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 -- 172.21.15.23:0/4164315390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102a30 msgr2=0x7f0c781a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102a30 0x7f0c781a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 -- 172.21.15.23:0/4164315390 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70009d00 con 0x7f0c78100f10 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7c8be700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c9e0 tx=0x7f0c6c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:29:11.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c007ac0 con 0x7f0c78100f10 2024-03-16T20:29:11.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c6c007c20 con 0x7f0c78100f10 2024-03-16T20:29:11.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781a2330 con 0x7f0c78100f10 2024-03-16T20:29:11.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.333+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c018640 con 0x7f0c78100f10 2024-03-16T20:29:11.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.334+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c781a2830 con 0x7f0c78100f10 2024-03-16T20:29:11.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.335+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c78068ca0 con 0x7f0c78100f10 2024-03-16T20:29:11.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.336+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f0c6c00f450 con 0x7f0c78100f10 2024-03-16T20:29:11.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.336+0000 7f0c75ffb700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 0x7f0c680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:11.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.336+0000 7f0c77fff700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 0x7f0c680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:11.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.336+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f0c6c0cbe90 con 0x7f0c78100f10 2024-03-16T20:29:11.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.337+0000 7f0c77fff700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 0x7f0c680a3ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c64005950 tx=0x7f0c6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:29:11.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.341+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0c6c096560 con 0x7f0c78100f10 2024-03-16T20:29:11.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:11.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:11.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:11 smithi031 ceph-mon[36281]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T20:29:11.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:11 smithi189 ceph-mon[36655]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T20:29:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:11.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:11.555+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c780fe580 con 0x7f0c680a1a30 2024-03-16T20:29:11.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:11.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:11.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:11 smithi023 ceph-mon[31879]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T20:29:11.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T20:29:12.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:12 smithi023 ceph-mon[31879]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:12.404 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48806]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:12.404 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48806]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:12.404 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48806]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:12.404 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48806]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:12.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T20:29:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:12 smithi189 ceph-mon[36655]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:29:12.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T20:29:12.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:12 smithi031 ceph-mon[36281]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:12.828 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48826]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:12.828 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48826]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:12.828 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48826]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:12.828 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:29:12 smithi023 sudo[48826]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:13.187 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:12 smithi031 sudo[44589]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:13.187 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:12 smithi031 sudo[44589]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:13.187 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:12 smithi031 sudo[44589]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:13.187 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:12 smithi031 sudo[44589]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:13.444+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0c6c096930 con 0x7f0c78100f10 2024-03-16T20:29:13.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:13 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:13.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:13 smithi031 ceph-mon[36281]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T20:29:13.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:13 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:13.688 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:13 smithi031 sudo[44592]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:13.688 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:13 smithi031 sudo[44592]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:13.688 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:13 smithi031 sudo[44592]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:13.688 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:29:13 smithi031 sudo[44592]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:13.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:13 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:13.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:13 smithi189 ceph-mon[36655]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T20:29:13.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:13 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:13.720 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:13.720 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:13.720 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45707]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:13.720 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45707]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:13.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:13 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:13.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:13 smithi023 ceph-mon[31879]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T20:29:13.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:13 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:14.219 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:14.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:14.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:14.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:29:13 smithi189 sudo[45710]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48895]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48895]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48895]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48895]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:14.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48898]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:14.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48898]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:14.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48898]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:14.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 sudo[48898]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:14.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45713]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:14.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45713]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:14.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45713]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:14.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45713]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:15.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T20:29:15.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45716]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45716]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 sudo[45716]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44595]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T20:29:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44595]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44595]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44595]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:16.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44598]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T20:29:16.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44598]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T20:29:16.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44598]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 sudo[44598]: pam_unix(sudo:session): session closed for user root 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]': finished 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T20:29:16.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:15 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]': finished 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T20:29:16.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:15 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1411823731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc0249a5-38ab-46a3-957f-f13367ec3060"}]': finished 2024-03-16T20:29:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T20:29:16.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:15 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:17.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:16 smithi031 ceph-mon[36281]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:17.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:16 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2236262645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:17.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:16 smithi189 ceph-mon[36655]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:17.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:16 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2236262645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:17.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:16 smithi023 ceph-mon[31879]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:17.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:16 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2236262645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:19.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:18 smithi031 ceph-mon[36281]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:19.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:18 smithi189 ceph-mon[36655]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:19.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:18 smithi023 ceph-mon[31879]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:20.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:20 smithi023 ceph-mon[31879]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:21.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:20 smithi031 ceph-mon[36281]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:21.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:20 smithi189 ceph-mon[36655]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:23.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:22 smithi023 ceph-mon[31879]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:23.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:22 smithi031 ceph-mon[36281]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:23.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:22 smithi189 ceph-mon[36655]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:24.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T20:29:24.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:24.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T20:29:24.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:24.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T20:29:24.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:25.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:24 smithi023 ceph-mon[31879]: Deploying daemon osd.3 on smithi023 2024-03-16T20:29:25.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:24 smithi023 ceph-mon[31879]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:25.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:24 smithi031 ceph-mon[36281]: Deploying daemon osd.3 on smithi023 2024-03-16T20:29:25.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:24 smithi031 ceph-mon[36281]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:25.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:24 smithi189 ceph-mon[36655]: Deploying daemon osd.3 on smithi023 2024-03-16T20:29:25.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:24 smithi189 ceph-mon[36655]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:27.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:26 smithi023 ceph-mon[31879]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:27.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:26 smithi031 ceph-mon[36281]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:27.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:26 smithi189 ceph-mon[36655]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:29.187 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:28 smithi023 ceph-mon[31879]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:29.187 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.187 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.187 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:28 smithi031 ceph-mon[36281]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:28 smithi189 ceph-mon[36655]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:31.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:30 smithi023 ceph-mon[31879]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:31.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:30 smithi031 ceph-mon[36281]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:31.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:30 smithi189 ceph-mon[36655]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:31.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.361+0000 7f0c75ffb700 1 -- 172.21.15.23:0/4164315390 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0c780fe580 con 0x7f0c680a1a30 2024-03-16T20:29:31.365 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 msgr2=0x7f0c680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 0x7f0c680a3ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c64005950 tx=0x7f0c6400b410 comp rx=0 tx=0).stop 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 msgr2=0x7f0c7819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c9e0 tx=0x7f0c6c00ccf0 comp rx=0 tx=0).stop 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 shutdown_connections 2024-03-16T20:29:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0c680a1a30 0x7f0c680a3ef0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c78100f10 0x7f0c7819a1f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c78102a30 0x7f0c781a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 --2- 172.21.15.23:0/4164315390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c78102070 0x7f0c7819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 >> 172.21.15.23:0/4164315390 conn(0x7f0c780fc4b0 msgr2=0x7f0c780fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 shutdown_connections 2024-03-16T20:29:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:29:31.363+0000 7f0c7eb22700 1 -- 172.21.15.23:0/4164315390 wait complete. 2024-03-16T20:29:31.987 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T20:29:31.990 INFO:tasks.cephadm:Deploying osd.4 on smithi031 with /dev/vg_nvme/lv_3... 2024-03-16T20:29:31.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T20:29:32.249 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:29:32.314 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:29:32.195+0000 7f43aea7d7c0 -1 osd.3 0 log_to_monitors true 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.566 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:32 smithi031 ceph-mon[36281]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:32 smithi023 ceph-mon[31879]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:32 smithi189 ceph-mon[36655]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T20:29:34.199 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:34 smithi031 ceph-mon[36281]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T20:29:34.199 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:34 smithi031 ceph-mon[36281]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T20:29:34.200 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:34 smithi031 ceph-mon[36281]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:29:34.200 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:34.200 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:34 smithi031 ceph-mon[36281]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:34.200 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T20:29:34.200 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T20:29:34.200 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-16T20:29:34.200 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-16T20:29:34.200 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110694 s, 947 MB/s 2024-03-16T20:29:34.201 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-16T20:29:34.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-mon[31879]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T20:29:34.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-mon[31879]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T20:29:34.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-mon[31879]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:29:34.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:34.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-mon[31879]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:34 smithi189 ceph-mon[36655]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T20:29:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:34 smithi189 ceph-mon[36655]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T20:29:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:34 smithi189 ceph-mon[36655]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:29:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:34 smithi189 ceph-mon[36655]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:34.839 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:29:34.563+0000 7f43a40e2700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T20:29:34.839 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:29:34 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:29:34.569+0000 7f439becd700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:29:34.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-03-16T20:29:35.119 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:29:35.418 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:35 smithi031 ceph-mon[36281]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:29:35.418 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:35 smithi031 ceph-mon[36281]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T20:29:35.418 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:35 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.418 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:35 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.418 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:35 smithi031 ceph-mon[36281]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' 2024-03-16T20:29:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:35 smithi189 ceph-mon[36655]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:29:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:35 smithi189 ceph-mon[36655]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T20:29:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:35 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:35 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:35 smithi189 ceph-mon[36655]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' 2024-03-16T20:29:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:35 smithi023 ceph-mon[31879]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:29:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:35 smithi023 ceph-mon[31879]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T20:29:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:35 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:35 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:35 smithi023 ceph-mon[31879]: from='osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291]' entity='osd.3' 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] boot 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi023 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.173 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.174 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:36 smithi031 ceph-mon[36281]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] boot 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T20:29:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi023 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:36 smithi189 ceph-mon[36655]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:36.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:29:36.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:29:36.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: osd.3 [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] boot 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi023 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:36.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:36.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:29:36.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:36.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:36 smithi023 ceph-mon[31879]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-16T20:29:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 -- 172.21.15.31:0/3755574022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 msgr2=0x7f506c1024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 --2- 172.21.15.31:0/3755574022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1024f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5060009990 tx=0x7f5060009ca0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 -- 172.21.15.31:0/3755574022 shutdown_connections 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 --2- 172.21.15.31:0/3755574022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c100f10 0x7f506c101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 --2- 172.21.15.31:0/3755574022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c102a30 0x7f506c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 --2- 172.21.15.31:0/3755574022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1024f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 -- 172.21.15.31:0/3755574022 >> 172.21.15.31:0/3755574022 conn(0x7f506c0fc4b0 msgr2=0x7f506c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 -- 172.21.15.31:0/3755574022 shutdown_connections 2024-03-16T20:29:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.704+0000 7f5072359700 1 -- 172.21.15.31:0/3755574022 wait complete. 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 Processor -- start 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 -- start start 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c100f10 0x7f506c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c102a30 0x7f506c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c19faf0 con 0x7f506c102a30 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c19fc30 con 0x7f506c102070 2024-03-16T20:29:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f5072359700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c19fdc0 con 0x7f506c100f10 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33796/0 (socket says 172.21.15.31:33796) 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f506b7fe700 1 -- 172.21.15.31:0/3551488786 learned_addr learned my addr 172.21.15.31:0/3551488786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f50708f6700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c102a30 0x7f506c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.705+0000 7f506bfff700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c100f10 0x7f506c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 -- 172.21.15.31:0/3551488786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c100f10 msgr2=0x7f506c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c100f10 0x7f506c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 -- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c102a30 msgr2=0x7f506c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c102a30 0x7f506c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 -- 172.21.15.31:0/3551488786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600095f0 con 0x7f506c102070 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f506b7fe700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5060006010 tx=0x7f5060004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:29:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506001d070 con 0x7f506c102070 2024-03-16T20:29:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c1a0040 con 0x7f506c102070 2024-03-16T20:29:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5060005030 con 0x7f506c102070 2024-03-16T20:29:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060004d90 con 0x7f506c102070 2024-03-16T20:29:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.706+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c1a0450 con 0x7f506c102070 2024-03-16T20:29:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.707+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c068ca0 con 0x7f506c102070 2024-03-16T20:29:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.708+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f50600051a0 con 0x7f506c102070 2024-03-16T20:29:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.709+0000 7f50697fa700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 0x7f50540a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:29:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.709+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f50600cd890 con 0x7f506c102070 2024-03-16T20:29:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.709+0000 7f506bfff700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 0x7f50540a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:29:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.709+0000 7f506bfff700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 0x7f50540a3f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f505c009c60 tx=0x7f505c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:29:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.713+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5060097d30 con 0x7f506c102070 2024-03-16T20:29:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:36.931+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f506c0fe650 con 0x7f50540a1ac0 2024-03-16T20:29:37.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:37 smithi031 ceph-mon[36281]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T20:29:37.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:37 smithi031 ceph-mon[36281]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:37.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:37.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:37.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:37 smithi189 ceph-mon[36655]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T20:29:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:37 smithi189 ceph-mon[36655]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:37 smithi023 ceph-mon[31879]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T20:29:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:37 smithi023 ceph-mon[31879]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:29:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:29:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:29:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:38.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:38 smithi031 ceph-mon[36281]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T20:29:38.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:38 smithi031 ceph-mon[36281]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:38 smithi189 ceph-mon[36655]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T20:29:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:38 smithi189 ceph-mon[36655]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:38 smithi023 ceph-mon[31879]: osdmap e27: 4 total, 4 up, 4 in 2024-03-16T20:29:38.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:38 smithi023 ceph-mon[31879]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:39.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:39 smithi031 ceph-mon[36281]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T20:29:39.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:39 smithi189 ceph-mon[36655]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T20:29:39.498 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:39 smithi023 ceph-mon[31879]: osdmap e28: 4 total, 4 up, 4 in 2024-03-16T20:29:40.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T20:29:40.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/3145315902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]': finished 2024-03-16T20:29:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T20:29:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:40 smithi031 ceph-mon[36281]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T20:29:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/3145315902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]': finished 2024-03-16T20:29:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T20:29:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:40 smithi189 ceph-mon[36655]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:40.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: osdmap e29: 4 total, 4 up, 4 in 2024-03-16T20:29:40.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/3145315902' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]: dispatch 2024-03-16T20:29:40.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ff34b3-508e-49ca-a5aa-61c2c52551b1"}]': finished 2024-03-16T20:29:40.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: osdmap e30: 5 total, 4 up, 5 in 2024-03-16T20:29:40.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:40.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:40 smithi023 ceph-mon[31879]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:41.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:41 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/152596335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:41.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:41 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/152596335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:41 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/152596335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:29:42.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:42 smithi189 ceph-mon[36655]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T20:29:42.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:42 smithi023 ceph-mon[31879]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T20:29:42.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:42 smithi031 ceph-mon[36281]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-16T20:29:44.067 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:43 smithi031 ceph-mon[36281]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:29:44.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:43 smithi023 ceph-mon[31879]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:29:44.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:43 smithi189 ceph-mon[36655]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:29:46.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:45 smithi031 ceph-mon[36281]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T20:29:46.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:45 smithi023 ceph-mon[31879]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T20:29:46.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:45 smithi189 ceph-mon[36655]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-16T20:29:47.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T20:29:47.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:47.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:46 smithi023 ceph-mon[31879]: Deploying daemon osd.4 on smithi031 2024-03-16T20:29:47.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T20:29:47.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:47.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:46 smithi031 ceph-mon[36281]: Deploying daemon osd.4 on smithi031 2024-03-16T20:29:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T20:29:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:29:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:46 smithi189 ceph-mon[36655]: Deploying daemon osd.4 on smithi031 2024-03-16T20:29:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:47 smithi031 ceph-mon[36281]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-16T20:29:48.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:47 smithi023 ceph-mon[31879]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-16T20:29:48.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:47 smithi189 ceph-mon[36655]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-16T20:29:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:49 smithi031 ceph-mon[36281]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-16T20:29:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:49 smithi023 ceph-mon[31879]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-16T20:29:50.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:49 smithi189 ceph-mon[36655]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-16T20:29:52.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:51 smithi031 ceph-mon[36281]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T20:29:52.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:51 smithi023 ceph-mon[31879]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T20:29:52.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:51 smithi189 ceph-mon[36655]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-16T20:29:53.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:53.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:53.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:29:54.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:54 smithi031 ceph-mon[36281]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:54.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:54 smithi189 ceph-mon[36655]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:54.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:54 smithi023 ceph-mon[31879]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:55.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.946+0000 7f50697fa700 1 -- 172.21.15.31:0/3551488786 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f506c0fe650 con 0x7f50540a1ac0 2024-03-16T20:29:55.950 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 4 on host 'smithi031' 2024-03-16T20:29:55.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 msgr2=0x7f50540a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:55.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 0x7f50540a3f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f505c009c60 tx=0x7f505c009260 comp rx=0 tx=0).stop 2024-03-16T20:29:55.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 msgr2=0x7f506c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:29:55.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5060006010 tx=0x7f5060004730 comp rx=0 tx=0).stop 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 shutdown_connections 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50540a1ac0 0x7f50540a3f80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c102a30 0x7f506c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f506c100f10 0x7f506c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 --2- 172.21.15.31:0/3551488786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f506c102070 0x7f506c1984a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 >> 172.21.15.31:0/3551488786 conn(0x7f506c0fc4b0 msgr2=0x7f506c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 shutdown_connections 2024-03-16T20:29:55.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:29:55.948+0000 7f5072359700 1 -- 172.21.15.31:0/3551488786 wait complete. 2024-03-16T20:29:56.562 DEBUG:teuthology.orchestra.run.smithi031:osd.4> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T20:29:56.564 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_3... 2024-03-16T20:29:56.565 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-16T20:29:56.812 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:56 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:56 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:56 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.039 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:57 smithi189 ceph-mon[36655]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.235 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-mon[36281]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:58.236 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4[47392]: 2024-03-16T20:29:57.962+0000 7fe4b4c3d700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T20:29:58.236 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:29:57 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4[47392]: 2024-03-16T20:29:57.968+0000 7fe4b0a30700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:29:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T20:29:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: osdmap e31: 5 total, 4 up, 5 in 2024-03-16T20:29:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: from='osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:29:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:57 smithi023 ceph-mon[31879]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:29:58.953 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-16T20:29:58.953 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-16T20:29:58.954 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-16T20:29:58.954 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-16T20:29:58.954 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112883 s, 929 MB/s 2024-03-16T20:29:58.954 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-16T20:29:58.979 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:58 smithi189 ceph-mon[36655]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:29:58.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:58 smithi189 ceph-mon[36655]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T20:29:58.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:58.980 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:59.232 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:58 smithi031 ceph-mon[36281]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:29:59.232 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:58 smithi031 ceph-mon[36281]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T20:29:59.232 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:59.232 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:59.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:58 smithi023 ceph-mon[31879]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:29:59.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:58 smithi023 ceph-mon[31879]: osdmap e32: 5 total, 4 up, 5 in 2024-03-16T20:29:59.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:59.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:29:59.625 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-03-16T20:29:59.862 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] boot 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi031 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi031 to 8542M 2024-03-16T20:30:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:00.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:00.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:29:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] boot 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi031 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.230 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi031 to 8542M 2024-03-16T20:30:00.231 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.231 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:00.231 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:00.231 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:29:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:30:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:30:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: osd.4 [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] boot 2024-03-16T20:30:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi031 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi031 to 8542M 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:29:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: overall HEALTH_OK 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: overall HEALTH_OK 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:01.228 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T20:30:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: overall HEALTH_OK 2024-03-16T20:30:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:01.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:01.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 -- 172.21.15.189:0/271521831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 msgr2=0x7f41f8103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:01.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 --2- 172.21.15.189:0/271521831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f8103670 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009990 tx=0x7f41e8009ca0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.469 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 -- 172.21.15.189:0/271521831 shutdown_connections 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 --2- 172.21.15.189:0/271521831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8102010 0x7f41f8102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 --2- 172.21.15.189:0/271521831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8103bb0 0x7f41f810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 --2- 172.21.15.189:0/271521831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f8103670 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.465+0000 7f41fcd73700 1 -- 172.21.15.189:0/271521831 >> 172.21.15.189:0/271521831 conn(0x7f41f80fd5f0 msgr2=0x7f41f80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 -- 172.21.15.189:0/271521831 shutdown_connections 2024-03-16T20:30:01.470 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 -- 172.21.15.189:0/271521831 wait complete. 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 Processor -- start 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 -- start start 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8102010 0x7f41f8075d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.466+0000 7f41fcd73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 0x7f41f8074910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80750b0 con 0x7f41f8103bb0 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8075220 con 0x7f41f8102010 2024-03-16T20:30:01.472 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f81a2090 con 0x7f41f8103210 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:43794/0 (socket says 172.21.15.189:43794) 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 -- 172.21.15.189:0/3855361214 learned_addr learned my addr 172.21.15.189:0/3855361214 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f659c700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8102010 0x7f41f8075d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f6d9d700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 0x7f41f8074910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 -- 172.21.15.189:0/3855361214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8102010 msgr2=0x7f41f8075d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8102010 0x7f41f8075d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 -- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 msgr2=0x7f41f8074910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 0x7f41f8074910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 -- 172.21.15.189:0/3855361214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80095f0 con 0x7f41f8103210 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f6d9d700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 0x7f41f8074910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41f5d9b700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41e8003930 tx=0x7f41e8003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:01.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e801d070 con 0x7f41f8103210 2024-03-16T20:30:01.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f81a22c0 con 0x7f41f8103210 2024-03-16T20:30:01.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41e800f460 con 0x7f41f8103210 2024-03-16T20:30:01.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8017650 con 0x7f41f8103210 2024-03-16T20:30:01.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.467+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f81a2730 con 0x7f41f8103210 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.469+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f81093d0 con 0x7f41f8103210 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.470+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f41e800f5d0 con 0x7f41f8103210 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.470+0000 7f41e77fe700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 0x7f41e00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.470+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f41e80cd430 con 0x7f41f8103210 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.470+0000 7f41f659c700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 0x7f41e00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:01.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.471+0000 7f41f659c700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 0x7f41e00a4000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f41f8103070 tx=0x7f41ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:01.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.475+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f41e8097740 con 0x7f41f8103210 2024-03-16T20:30:01.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:01.675+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f41f8062420 con 0x7f41e00a1b40 2024-03-16T20:30:02.285 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:01 smithi189 ceph-mon[36655]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:02.286 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:01 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:02.286 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:01 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:02.286 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:01 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:02.286 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:01 smithi189 ceph-mon[36655]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:01 smithi023 ceph-mon[31879]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:01 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:01 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:01 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:01 smithi023 ceph-mon[31879]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:01 smithi031 ceph-mon[36281]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:01 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:01 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:01 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:01 smithi031 ceph-mon[36281]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:03 smithi023 ceph-mon[31879]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:04.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:03 smithi031 ceph-mon[36281]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:04.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:03 smithi189 ceph-mon[36655]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:04 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/3644717842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:04 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:04 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]': finished 2024-03-16T20:30:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:04 smithi189 ceph-mon[36655]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T20:30:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:04 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:04 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/3644717842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:04 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:04 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]': finished 2024-03-16T20:30:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:04 smithi023 ceph-mon[31879]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T20:30:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:04 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:04 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/3644717842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:04 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]: dispatch 2024-03-16T20:30:05.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:04 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b28071d-a593-49b1-9561-614bf17df712"}]': finished 2024-03-16T20:30:05.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:04 smithi031 ceph-mon[36281]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T20:30:05.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:04 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/2289070016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:05 smithi023 ceph-mon[31879]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:06.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/2289070016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:06.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:05 smithi031 ceph-mon[36281]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/2289070016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:05 smithi189 ceph-mon[36655]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:07 smithi023 ceph-mon[31879]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:08.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:07 smithi031 ceph-mon[36281]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:08.449 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:07 smithi189 ceph-mon[36655]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:10.331 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:09 smithi189 ceph-mon[36655]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:10.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:09 smithi023 ceph-mon[31879]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:10.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:09 smithi031 ceph-mon[36281]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:11.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T20:30:11.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:12.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T20:30:12.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:12.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T20:30:12.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:12.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:12 smithi031 ceph-mon[36281]: Deploying daemon osd.5 on smithi189 2024-03-16T20:30:12.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:12 smithi031 ceph-mon[36281]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:12.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:12 smithi189 ceph-mon[36655]: Deploying daemon osd.5 on smithi189 2024-03-16T20:30:12.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:12 smithi189 ceph-mon[36655]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:13.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:12 smithi023 ceph-mon[31879]: Deploying daemon osd.5 on smithi189 2024-03-16T20:30:13.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:12 smithi023 ceph-mon[31879]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:14.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:13 smithi023 ceph-mon[31879]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:14.342 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:13 smithi189 ceph-mon[36655]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:14.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:13 smithi031 ceph-mon[36281]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:15 smithi023 ceph-mon[31879]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:15 smithi031 ceph-mon[36281]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:15 smithi189 ceph-mon[36655]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:18 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:18 smithi189 ceph-mon[36655]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:18.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:18 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:18.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:18 smithi031 ceph-mon[36281]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:18.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:18.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:18 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:18.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:18 smithi023 ceph-mon[31879]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:20.135 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:19 smithi189 ceph-mon[36655]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:19 smithi023 ceph-mon[31879]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:19 smithi031 ceph-mon[36281]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-03-16T20:30:21.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.246+0000 7f41e77fe700 1 -- 172.21.15.189:0/3855361214 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f41f8062420 con 0x7f41e00a1b40 2024-03-16T20:30:21.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.248+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 msgr2=0x7f41e00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.248+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 0x7f41e00a4000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f41f8103070 tx=0x7f41ec00b410 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.248+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 msgr2=0x7f41f80743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.248+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41e8003930 tx=0x7f41e8003c00 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 shutdown_connections 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f41e00a1b40 0x7f41e00a4000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f8103bb0 0x7f41f8074910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41f8102010 0x7f41f8075d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 --2- 172.21.15.189:0/3855361214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41f8103210 0x7f41f80743d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 >> 172.21.15.189:0/3855361214 conn(0x7f41f80fd5f0 msgr2=0x7f41f8106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 shutdown_connections 2024-03-16T20:30:21.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:30:21.249+0000 7f41fcd73700 1 -- 172.21.15.189:0/3855361214 wait complete. 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:21 smithi189 ceph-mon[36655]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:21 smithi031 ceph-mon[36281]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:21 smithi023 ceph-mon[31879]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T20:30:21.943 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T20:30:21.946 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-16T20:30:21.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T20:30:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:30:22.455 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:22.455 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T20:30:22.455 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T20:30:22.455 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:22.455 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:22.456 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:22 smithi189 ceph-mon[36655]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:22.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:22 smithi023 ceph-mon[31879]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:22.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:22.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T20:30:22.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T20:30:22.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:22.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: from='osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:22.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:22 smithi031 ceph-mon[36281]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:30:23.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:30:23 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5[48588]: 2024-03-16T20:30:23.258+0000 7f8bc85c1700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T20:30:23.367 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:30:23 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5[48588]: 2024-03-16T20:30:23.264+0000 7f8bc43b4700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:30:24.185 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T20:30:24.185 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T20:30:24.185 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-16T20:30:24.185 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-16T20:30:24.185 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148843 s, 704 MB/s 2024-03-16T20:30:24.186 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-16T20:30:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:30:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T20:30:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] boot 2024-03-16T20:30:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T20:30:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:24 smithi189 ceph-mon[36655]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] boot 2024-03-16T20:30:24.489 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T20:30:24.490 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.490 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:24 smithi023 ceph-mon[31879]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: osd.5 [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] boot 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T20:30:24.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:24 smithi031 ceph-mon[36281]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-16T20:30:24.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-16T20:30:25.083 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi189 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-16T20:30:25.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:25.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:25.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi189 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:25.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:30:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:30:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T20:30:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi189 2024-03-16T20:30:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 -- 172.21.15.23:0/4170528092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6418102a90 msgr2=0x7f64181093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 --2- 172.21.15.23:0/4170528092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6418102a90 0x7f64181093b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f641000b0a0 tx=0x7f641000b3b0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 -- 172.21.15.23:0/4170528092 shutdown_connections 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 --2- 172.21.15.23:0/4170528092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6418102a90 0x7f64181093b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 --2- 172.21.15.23:0/4170528092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418100ef0 0x7f6418101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 --2- 172.21.15.23:0/4170528092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64181020d0 0x7f6418102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 -- 172.21.15.23:0/4170528092 >> 172.21.15.23:0/4170528092 conn(0x7f64180fc500 msgr2=0x7f64180fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 -- 172.21.15.23:0/4170528092 shutdown_connections 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.674+0000 7f641d39a700 1 -- 172.21.15.23:0/4170528092 wait complete. 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 Processor -- start 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 -- start start 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6418100ef0 0x7f6418104f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418102a90 0x7f6418103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6418104270 con 0x7f64181020d0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181043e0 con 0x7f6418100ef0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f641d39a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181a21b0 con 0x7f6418102a90 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59452/0 (socket says 172.21.15.23:59452) 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 -- 172.21.15.23:0/1265752693 learned_addr learned my addr 172.21.15.23:0/1265752693 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f6416ffd700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6418100ef0 0x7f6418104f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64177fe700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418102a90 0x7f6418103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 -- 172.21.15.23:0/1265752693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418102a90 msgr2=0x7f6418103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418102a90 0x7f6418103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 -- 172.21.15.23:0/1265752693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6418100ef0 msgr2=0x7f6418104f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6418100ef0 0x7f6418104f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.675+0000 7f64167fc700 1 -- 172.21.15.23:0/1265752693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410009d00 con 0x7f64181020d0 2024-03-16T20:30:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f64167fc700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f640800d950 tx=0x7f640800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:26.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408009940 con 0x7f64181020d0 2024-03-16T20:30:26.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64181a2490 con 0x7f64181020d0 2024-03-16T20:30:26.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6408010460 con 0x7f64181020d0 2024-03-16T20:30:26.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640800f620 con 0x7f64181020d0 2024-03-16T20:30:26.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.676+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64181a2a60 con 0x7f64181020d0 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.677+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6418068ca0 con 0x7f64181020d0 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.679+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f64080105d0 con 0x7f64181020d0 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.679+0000 7f640ffff700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 0x7f64040a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.679+0000 7f6416ffd700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 0x7f64040a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.679+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f64080cc400 con 0x7f64181020d0 2024-03-16T20:30:26.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.680+0000 7f6416ffd700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 0x7f64040a3f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6418100c20 tx=0x7f6400009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:26.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.683+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6408099fe0 con 0x7f64181020d0 2024-03-16T20:30:26.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:26.903+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f641806ae30 con 0x7f64040a1ac0 2024-03-16T20:30:27.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:26 smithi031 ceph-mon[36281]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T20:30:27.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:26 smithi031 ceph-mon[36281]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:27.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:26 smithi189 ceph-mon[36655]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T20:30:27.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:26 smithi189 ceph-mon[36655]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:27.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:26 smithi023 ceph-mon[31879]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T20:30:27.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:26 smithi023 ceph-mon[31879]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:28.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:27 smithi031 ceph-mon[36281]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T20:30:28.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:27 smithi031 ceph-mon[36281]: from='client.14379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:28.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:28.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:28.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:27 smithi189 ceph-mon[36655]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T20:30:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:27 smithi189 ceph-mon[36655]: from='client.14379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:27 smithi023 ceph-mon[31879]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T20:30:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:27 smithi023 ceph-mon[31879]: from='client.14379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:28.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:29.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:28 smithi023 ceph-mon[31879]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T20:30:29.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:28 smithi023 ceph-mon[31879]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:30:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:28 smithi031 ceph-mon[36281]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T20:30:29.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:28 smithi031 ceph-mon[36281]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:30:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:28 smithi189 ceph-mon[36655]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T20:30:29.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:28 smithi189 ceph-mon[36655]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:30:30.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:29 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]: dispatch 2024-03-16T20:30:30.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:29 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]': finished 2024-03-16T20:30:30.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:29 smithi031 ceph-mon[36281]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T20:30:30.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:29 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:30.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:29 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]: dispatch 2024-03-16T20:30:30.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:29 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]': finished 2024-03-16T20:30:30.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:29 smithi189 ceph-mon[36655]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T20:30:30.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:29 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:29 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]: dispatch 2024-03-16T20:30:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:29 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4226776749' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7698cf3-f3a6-4e39-991a-b101bc81d7ec"}]': finished 2024-03-16T20:30:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:29 smithi023 ceph-mon[31879]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T20:30:30.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:29 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:31.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:30 smithi031 ceph-mon[36281]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-16T20:30:31.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:30 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3278286591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:31.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:30 smithi189 ceph-mon[36655]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-16T20:30:31.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:30 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3278286591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:30 smithi023 ceph-mon[31879]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-16T20:30:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:30 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3278286591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:33.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:32 smithi031 ceph-mon[36281]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:30:33.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:32 smithi189 ceph-mon[36655]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:30:33.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:32 smithi023 ceph-mon[31879]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:30:35.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:34 smithi031 ceph-mon[36281]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:35.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:34 smithi189 ceph-mon[36655]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:35.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:34 smithi023 ceph-mon[31879]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:37.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:36 smithi023 ceph-mon[31879]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:37.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:36 smithi031 ceph-mon[36281]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:37.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:36 smithi189 ceph-mon[36655]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:37.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T20:30:37.938 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:38.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T20:30:38.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:38.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T20:30:38.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:39.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:38 smithi023 ceph-mon[31879]: Deploying daemon osd.6 on smithi023 2024-03-16T20:30:39.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:38 smithi023 ceph-mon[31879]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:39.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:38 smithi189 ceph-mon[36655]: Deploying daemon osd.6 on smithi023 2024-03-16T20:30:39.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:38 smithi189 ceph-mon[36655]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:39.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:38 smithi031 ceph-mon[36281]: Deploying daemon osd.6 on smithi023 2024-03-16T20:30:39.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:38 smithi031 ceph-mon[36281]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:41.192 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:40 smithi023 ceph-mon[31879]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:41.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:40 smithi189 ceph-mon[36655]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:41.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:40 smithi031 ceph-mon[36281]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:43.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:42 smithi189 ceph-mon[36655]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:43.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:43.306 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:42 smithi023 ceph-mon[31879]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:43.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:42 smithi031 ceph-mon[36281]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:45.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:44 smithi023 ceph-mon[31879]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:45.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:44 smithi189 ceph-mon[36655]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:45.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:44 smithi031 ceph-mon[36281]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:46.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:45 smithi189 ceph-mon[36655]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:46.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:45 smithi023 ceph-mon[31879]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:46.330 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2024-03-16T20:30:46.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.325+0000 7f640ffff700 1 -- 172.21.15.23:0/1265752693 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f641806ae30 con 0x7f64040a1ac0 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.328+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 msgr2=0x7f64040a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.328+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 0x7f64040a3f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6418100c20 tx=0x7f6400009190 comp rx=0 tx=0).stop 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 msgr2=0x7f64181035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f640800d950 tx=0x7f640800dc60 comp rx=0 tx=0).stop 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 shutdown_connections 2024-03-16T20:30:46.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64040a1ac0 0x7f64040a3f80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6418102a90 0x7f6418103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6418100ef0 0x7f6418104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 --2- 172.21.15.23:0/1265752693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64181020d0 0x7f64181035b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 >> 172.21.15.23:0/1265752693 conn(0x7f64180fc500 msgr2=0x7f641806a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 shutdown_connections 2024-03-16T20:30:46.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:30:46.329+0000 7f641d39a700 1 -- 172.21.15.23:0/1265752693 wait complete. 2024-03-16T20:30:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:45 smithi031 ceph-mon[36281]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:46.951 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T20:30:46.953 INFO:tasks.cephadm:Deploying osd.7 on smithi031 with /dev/vg_nvme/lv_2... 2024-03-16T20:30:46.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T20:30:47.214 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:30:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:47 smithi023 ceph-mon[31879]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T20:30:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.627 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.627 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.627 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:47 smithi031 ceph-mon[36281]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T20:30:47.627 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.627 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:47 smithi189 ceph-mon[36655]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T20:30:47.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:47.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:48.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-mon[31879]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T20:30:48.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-mon[31879]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T20:30:48.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-mon[31879]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:30:48.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:48.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-mon[31879]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:48.590 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6[57485]: 2024-03-16T20:30:48.324+0000 7fc524b02700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T20:30:48.591 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:30:48 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6[57485]: 2024-03-16T20:30:48.337+0000 7fc5208f5700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:30:48.659 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:48 smithi031 ceph-mon[36281]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T20:30:48.659 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:48 smithi031 ceph-mon[36281]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:48 smithi031 ceph-mon[36281]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:48 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:48.660 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:48 smithi031 ceph-mon[36281]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:48 smithi189 ceph-mon[36655]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T20:30:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:48 smithi189 ceph-mon[36655]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T20:30:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:48 smithi189 ceph-mon[36655]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:30:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:48 smithi189 ceph-mon[36655]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T20:30:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T20:30:49.240 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-16T20:30:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-16T20:30:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123797 s, 847 MB/s 2024-03-16T20:30:49.240 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-16T20:30:49.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:49 smithi031 ceph-mon[36281]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:30:49.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:49 smithi031 ceph-mon[36281]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T20:30:49.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:49 smithi023 ceph-mon[31879]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:30:49.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:49 smithi023 ceph-mon[31879]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T20:30:49.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:49 smithi189 ceph-mon[36655]: from='osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:30:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:49 smithi189 ceph-mon[36655]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T20:30:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:49.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-03-16T20:30:50.106 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:30:50.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:30:50.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:30:50.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.538 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] boot 2024-03-16T20:30:50.539 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T20:30:50.539 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.539 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:50 smithi031 ceph-mon[36281]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] boot 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:50 smithi023 ceph-mon[31879]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:50.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:30:50.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:30:50.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: osd.6 [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] boot 2024-03-16T20:30:50.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T20:30:50.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T20:30:50.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:50 smithi189 ceph-mon[36655]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-16T20:30:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T20:30:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi023 2024-03-16T20:30:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- 172.21.15.31:0/2628276618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103bd0 msgr2=0x7f79b010a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 --2- 172.21.15.31:0/2628276618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103bd0 0x7f79b010a480 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f79a80099c0 tx=0x7f79a8009cd0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- 172.21.15.31:0/2628276618 shutdown_connections 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 --2- 172.21.15.31:0/2628276618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0102030 0x7f79b0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 --2- 172.21.15.31:0/2628276618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103bd0 0x7f79b010a480 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 --2- 172.21.15.31:0/2628276618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0103230 0x7f79b0103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- 172.21.15.31:0/2628276618 >> 172.21.15.31:0/2628276618 conn(0x7f79b00fd5f0 msgr2=0x7f79b00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- 172.21.15.31:0/2628276618 shutdown_connections 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- 172.21.15.31:0/2628276618 wait complete. 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 Processor -- start 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.633+0000 7f79b822d700 1 -- start start 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0102030 0x7f79b0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0103bd0 0x7f79b019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b019fb20 con 0x7f79b0103bd0 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b019fc60 con 0x7f79b0103230 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b019fdf0 con 0x7f79b0102030 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42812/0 (socket says 172.21.15.31:42812) 2024-03-16T20:30:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 -- 172.21.15.31:0/489476670 learned_addr learned my addr 172.21.15.31:0/489476670 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b67ca700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0103bd0 0x7f79b019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b5fc9700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0102030 0x7f79b0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 -- 172.21.15.31:0/489476670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0102030 msgr2=0x7f79b0197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0102030 0x7f79b0197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 -- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0103bd0 msgr2=0x7f79b019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0103bd0 0x7f79b019f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 -- 172.21.15.31:0/489476670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a80095f0 con 0x7f79b0103230 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b57c8700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79a400b530 tx=0x7f79a400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4011830 con 0x7f79b0103230 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.634+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b01a00d0 con 0x7f79b0103230 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.635+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a4011e70 con 0x7f79b0103230 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.635+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b01a0620 con 0x7f79b0103230 2024-03-16T20:30:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.635+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400f2e0 con 0x7f79b0103230 2024-03-16T20:30:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.635+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b0109390 con 0x7f79b0103230 2024-03-16T20:30:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.637+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f79a40105f0 con 0x7f79b0103230 2024-03-16T20:30:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.637+0000 7f79aeffd700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 0x7f79980a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:30:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.637+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f79a40cc6a0 con 0x7f79b0103230 2024-03-16T20:30:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.637+0000 7f79b5fc9700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 0x7f79980a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:30:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.638+0000 7f79b5fc9700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 0x7f79980a3f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79b0103090 tx=0x7f79a0009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:30:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.641+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f79a40965c0 con 0x7f79b0103230 2024-03-16T20:30:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T20:30:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi023 2024-03-16T20:30:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi023 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:30:51.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:51.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:30:51.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:30:51.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:30:51.853+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f79b002da30 con 0x7f79980a1ac0 2024-03-16T20:30:52.655 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:52 smithi031 ceph-mon[36281]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:52.655 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:52.655 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:52.655 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:52.656 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:52 smithi031 ceph-mon[36281]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:52 smithi189 ceph-mon[36655]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:52 smithi189 ceph-mon[36655]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:52 smithi023 ceph-mon[31879]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:30:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:30:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:30:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:30:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:52 smithi023 ceph-mon[31879]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:54.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:53 smithi023 ceph-mon[31879]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:53 smithi031 ceph-mon[36281]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:53 smithi189 ceph-mon[36655]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:55.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:54 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/3647204708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:54 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:54 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]': finished 2024-03-16T20:30:55.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:54 smithi023 ceph-mon[31879]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T20:30:55.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:30:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:54 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/3647204708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:54 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:54 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]': finished 2024-03-16T20:30:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:54 smithi031 ceph-mon[36281]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T20:30:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:54 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:30:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:54 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/3647204708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:55 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]: dispatch 2024-03-16T20:30:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:55 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82df9886-90ce-4a0a-8bdc-09b78caf46af"}]': finished 2024-03-16T20:30:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:55 smithi189 ceph-mon[36655]: osdmap e48: 8 total, 7 up, 8 in 2024-03-16T20:30:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:30:56.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/4122784040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:56.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:56 smithi023 ceph-mon[31879]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/4122784040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:56 smithi031 ceph-mon[36281]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/4122784040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:30:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:56 smithi189 ceph-mon[36655]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:57 smithi023 ceph-mon[31879]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:58.430 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:57 smithi031 ceph-mon[36281]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:30:58.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:57 smithi189 ceph-mon[36655]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:00.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:30:59 smithi031 ceph-mon[36281]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:00.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:30:59 smithi023 ceph-mon[31879]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:30:59 smithi189 ceph-mon[36655]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:02.273 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:01 smithi189 ceph-mon[36655]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:01 smithi023 ceph-mon[31879]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:01 smithi031 ceph-mon[36281]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:03.122 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T20:31:03.122 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:03.122 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:03 smithi031 ceph-mon[36281]: Deploying daemon osd.7 on smithi031 2024-03-16T20:31:03.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T20:31:03.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:03.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:03 smithi023 ceph-mon[31879]: Deploying daemon osd.7 on smithi031 2024-03-16T20:31:03.466 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T20:31:03.466 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:03.466 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:03 smithi189 ceph-mon[36655]: Deploying daemon osd.7 on smithi031 2024-03-16T20:31:04.104 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:04 smithi031 ceph-mon[36281]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:04.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:04 smithi023 ceph-mon[31879]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:04.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:04 smithi189 ceph-mon[36655]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:06.178 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:06 smithi031 ceph-mon[36281]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:06 smithi023 ceph-mon[31879]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:06 smithi189 ceph-mon[36655]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:08 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:08 smithi023 ceph-mon[31879]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:08 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:09.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:08 smithi031 ceph-mon[36281]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:09.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:08 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:09.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:08 smithi189 ceph-mon[36655]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:10.255 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:09 smithi031 ceph-mon[36281]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:10.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:09 smithi023 ceph-mon[31879]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:09 smithi189 ceph-mon[36655]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.271+0000 7f79aeffd700 1 -- 172.21.15.31:0/489476670 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f79b002da30 con 0x7f79980a1ac0 2024-03-16T20:31:11.277 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 7 on host 'smithi031' 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 msgr2=0x7f79980a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 0x7f79980a3f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79b0103090 tx=0x7f79a0009040 comp rx=0 tx=0).stop 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 msgr2=0x7f79b01984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79a400b530 tx=0x7f79a400b8f0 comp rx=0 tx=0).stop 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 shutdown_connections 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79980a1ac0 0x7f79980a3f80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0103bd0 0x7f79b019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:11.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b0102030 0x7f79b0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:11.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 --2- 172.21.15.31:0/489476670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b0103230 0x7f79b01984d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:11.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 >> 172.21.15.31:0/489476670 conn(0x7f79b00fd5f0 msgr2=0x7f79b01064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:31:11.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 shutdown_connections 2024-03-16T20:31:11.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:31:11.274+0000 7f79b822d700 1 -- 172.21.15.31:0/489476670 wait complete. 2024-03-16T20:31:11.859 DEBUG:teuthology.orchestra.run.smithi031:osd.7> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T20:31:11.861 INFO:tasks.cephadm:Deploying osd.8 on smithi189 with /dev/vg_nvme/lv_2... 2024-03-16T20:31:11.862 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-16T20:31:12.109 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:12 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:12 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:12.624 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:12 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:13.514 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:13 smithi189 ceph-mon[36655]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T20:31:13.514 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:13 smithi189 ceph-mon[36655]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T20:31:13.514 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:13 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:13.514 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:13 smithi189 ceph-mon[36655]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.514 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:13 smithi189 ceph-mon[36655]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.526 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-mon[36281]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T20:31:13.526 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-mon[36281]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T20:31:13.526 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:13.527 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-mon[36281]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.527 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-mon[36281]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:13 smithi023 ceph-mon[31879]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T20:31:13.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:13 smithi023 ceph-mon[31879]: osdmap e49: 8 total, 7 up, 8 in 2024-03-16T20:31:13.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:13 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:13.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:13 smithi023 ceph-mon[31879]: from='osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:13 smithi023 ceph-mon[31879]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:31:13.839 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7[54399]: 2024-03-16T20:31:13.717+0000 7fada983d700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T20:31:13.839 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:31:13 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7[54399]: 2024-03-16T20:31:13.727+0000 7fada1628700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:31:14.190 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112732 s, 930 MB/s 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 2024-03-16T20:31:14.191 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-16T20:31:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:31:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T20:31:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: from='osd.7 ' entity='osd.7' 2024-03-16T20:31:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] boot 2024-03-16T20:31:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T20:31:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:14 smithi031 ceph-mon[36281]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:14.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: from='osd.7 ' entity='osd.7' 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] boot 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:14 smithi189 ceph-mon[36655]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: osdmap e50: 8 total, 7 up, 8 in 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: from='osd.7 ' entity='osd.7' 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: osd.7 [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] boot 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: osdmap e51: 8 total, 8 up, 8 in 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T20:31:14.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:14 smithi023 ceph-mon[31879]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-16T20:31:14.848 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-03-16T20:31:15.089 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:31:15.414 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:15 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:31:15.414 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:15 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:31:15.414 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:15 smithi031 ceph-mon[36281]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T20:31:15.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:15 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:31:15.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:15 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:31:15.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:15 smithi189 ceph-mon[36655]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T20:31:15.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:15 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:31:15.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:15 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:31:15.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:15 smithi023 ceph-mon[31879]: osdmap e52: 8 total, 8 up, 8 in 2024-03-16T20:31:16.546 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi031 2024-03-16T20:31:16.546 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.546 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi031 to 5694M 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T20:31:16.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:16 smithi189 ceph-mon[36655]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 -- 172.21.15.189:0/154587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4100f10 msgr2=0x7f3bc4101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/154587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4100f10 0x7f3bc4101330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00098f0 tx=0x7f3bc0009c00 comp rx=0 tx=0).stop 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 -- 172.21.15.189:0/154587967 shutdown_connections 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/154587967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4102a30 0x7f3bc4109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/154587967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4100f10 0x7f3bc4101330 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/154587967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 -- 172.21.15.189:0/154587967 >> 172.21.15.189:0/154587967 conn(0x7f3bc40fc4d0 msgr2=0x7f3bc40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 -- 172.21.15.189:0/154587967 shutdown_connections 2024-03-16T20:31:16.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 -- 172.21.15.189:0/154587967 wait complete. 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.762+0000 7f3bcbda4700 1 Processor -- start 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 -- start start 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4100f10 0x7f3bc4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc419fb20 con 0x7f3bc4100f10 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc419fc60 con 0x7f3bc4102070 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc419fdf0 con 0x7f3bc4102a30 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:53814/0 (socket says 172.21.15.189:53814) 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 -- 172.21.15.189:0/548282579 learned_addr learned my addr 172.21.15.189:0/548282579 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bc9b40700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4100f10 0x7f3bc4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bc933f700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:16.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 -- 172.21.15.189:0/548282579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 msgr2=0x7f3bc41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 -- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4100f10 msgr2=0x7f3bc4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4100f10 0x7f3bc4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 -- 172.21.15.189:0/548282579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc0009520 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bc933f700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bca341700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc007f10 tx=0x7f3bbc00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc017070 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc41a00d0 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bbc00f040 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.763+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc005760 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.764+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc41a05d0 con 0x7f3bc4102a30 2024-03-16T20:31:16.772 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.765+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc4068ca0 con 0x7f3bc4102a30 2024-03-16T20:31:16.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.766+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3bbc0058c0 con 0x7f3bc4102a30 2024-03-16T20:31:16.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.767+0000 7f3bbaffd700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 0x7f3bb00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:16.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.767+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f3bbc0cc770 con 0x7f3bc4102a30 2024-03-16T20:31:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.767+0000 7f3bc9b40700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 0x7f3bb00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:16.774 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.768+0000 7f3bc9b40700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 0x7f3bb00a3f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009fd0 tx=0x7f3bc0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:31:16.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.770+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3bbc0964c0 con 0x7f3bc4102a30 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi031 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi031 to 5694M 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T20:31:16.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:16 smithi023 ceph-mon[31879]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:16.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi031 2024-03-16T20:31:16.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi031 to 5694M 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: osdmap e53: 8 total, 8 up, 8 in 2024-03-16T20:31:16.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:16 smithi031 ceph-mon[36281]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:16.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:16.978+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bc406ae10 con 0x7f3bb00a1ac0 2024-03-16T20:31:18.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:17 smithi031 ceph-mon[36281]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T20:31:18.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:17 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:18.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:17 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:18.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:17 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:18.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:17 smithi189 ceph-mon[36655]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T20:31:18.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:17 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:18.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:17 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:18.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:17 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:17 smithi023 ceph-mon[31879]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T20:31:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:19.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:18 smithi031 ceph-mon[36281]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:19.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:18 smithi031 ceph-mon[36281]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T20:31:19.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:18 smithi031 ceph-mon[36281]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:19.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:18 smithi189 ceph-mon[36655]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:19.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:18 smithi189 ceph-mon[36655]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T20:31:19.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:18 smithi189 ceph-mon[36655]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:19.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:18 smithi023 ceph-mon[31879]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:19.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:18 smithi023 ceph-mon[31879]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T20:31:19.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:18 smithi023 ceph-mon[31879]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:20.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:19 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/775277528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:19 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:19 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]': finished 2024-03-16T20:31:20.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:19 smithi031 ceph-mon[36281]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T20:31:20.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:20.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:19 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/775277528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:19 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:19 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]': finished 2024-03-16T20:31:20.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:19 smithi189 ceph-mon[36655]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T20:31:20.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:19 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/775277528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:19 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]: dispatch 2024-03-16T20:31:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:19 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49cd9654-b197-40a7-ab09-f7d720b898ff"}]': finished 2024-03-16T20:31:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:19 smithi023 ceph-mon[31879]: osdmap e56: 9 total, 8 up, 9 in 2024-03-16T20:31:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:21.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:20 smithi031 ceph-mon[36281]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:21.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:20 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/3233769057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:21.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:20 smithi189 ceph-mon[36655]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:21.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:20 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/3233769057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:21.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:20 smithi023 ceph-mon[31879]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:21.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:20 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/3233769057' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:23.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:22 smithi031 ceph-mon[36281]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:23.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:22 smithi189 ceph-mon[36655]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:23.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:22 smithi023 ceph-mon[31879]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:24.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:24 smithi189 ceph-mon[36655]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:25.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:24 smithi031 ceph-mon[36281]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:25.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:24 smithi023 ceph-mon[31879]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:27.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:26 smithi189 ceph-mon[36655]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:27.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:26 smithi023 ceph-mon[31879]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:27.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:26 smithi031 ceph-mon[36281]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T20:31:28.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:28.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T20:31:28.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:28.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-16T20:31:28.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:29.174 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:28 smithi189 ceph-mon[36655]: Deploying daemon osd.8 on smithi189 2024-03-16T20:31:29.174 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:28 smithi189 ceph-mon[36655]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:29.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:28 smithi023 ceph-mon[31879]: Deploying daemon osd.8 on smithi189 2024-03-16T20:31:29.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:28 smithi023 ceph-mon[31879]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:29.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:28 smithi031 ceph-mon[36281]: Deploying daemon osd.8 on smithi189 2024-03-16T20:31:29.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:28 smithi031 ceph-mon[36281]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:31.066 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:30 smithi189 ceph-mon[36655]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:31.241 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:30 smithi031 ceph-mon[36281]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:31.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:30 smithi023 ceph-mon[31879]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:32.257 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:32 smithi031 ceph-mon[36281]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:32.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:32 smithi189 ceph-mon[36655]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:32.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:32 smithi023 ceph-mon[31879]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:33.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:33.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:33.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:33 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:33.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:33 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:33.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:33 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:33.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:33 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:33 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:33 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:33 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:34.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:34 smithi189 ceph-mon[36655]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:34.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:34 smithi031 ceph-mon[36281]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:35.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:34 smithi023 ceph-mon[31879]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:36.501 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.496+0000 7f3bbaffd700 1 -- 172.21.15.189:0/548282579 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3bc406ae10 con 0x7f3bb00a1ac0 2024-03-16T20:31:36.501 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 8 on host 'smithi189' 2024-03-16T20:31:36.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 msgr2=0x7f3bb00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:36.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 0x7f3bb00a3f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009fd0 tx=0x7f3bc0019040 comp rx=0 tx=0).stop 2024-03-16T20:31:36.503 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 msgr2=0x7f3bc419f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc007f10 tx=0x7f3bbc00d3b0 comp rx=0 tx=0).stop 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 shutdown_connections 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3bb00a1ac0 0x7f3bb00a3f80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bc4100f10 0x7f3bc4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bc4102a30 0x7f3bc419f550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 --2- 172.21.15.189:0/548282579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bc4102070 0x7f3bc41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 >> 172.21.15.189:0/548282579 conn(0x7f3bc40fc4d0 msgr2=0x7f3bc406a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 shutdown_connections 2024-03-16T20:31:36.504 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:31:36.499+0000 7f3bcbda4700 1 -- 172.21.15.189:0/548282579 wait complete. 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:36.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:36 smithi189 ceph-mon[36655]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:36.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:36.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:36 smithi031 ceph-mon[36281]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:37.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:37.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:37.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:37.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:37.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:36 smithi023 ceph-mon[31879]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-16T20:31:37.153 DEBUG:teuthology.orchestra.run.smithi189:osd.8> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.8.service 2024-03-16T20:31:37.156 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-16T20:31:37.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T20:31:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:31:38.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T20:31:38.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T20:31:38.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:38.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-mon[36655]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:38.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T20:31:38.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T20:31:38.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:38.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:38 smithi023 ceph-mon[31879]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:38.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-16T20:31:38.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: from='osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: osdmap e57: 9 total, 8 up, 9 in 2024-03-16T20:31:38.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:38.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:31:38.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:38 smithi031 ceph-mon[36281]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:39.222 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8[55476]: 2024-03-16T20:31:38.917+0000 7f41ccab2700 -1 osd.8 0 waiting for initial osdmap 2024-03-16T20:31:39.222 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:31:38 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8[55476]: 2024-03-16T20:31:38.931+0000 7f41c489d700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:31:39.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:31:39.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T20:31:39.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T20:31:39.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: from='osd.8 ' entity='osd.8' 2024-03-16T20:31:39.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: from='osd.8 ' entity='osd.8' 2024-03-16T20:31:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120806 s, 868 MB/s 2024-03-16T20:31:39.784 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-16T20:31:39.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:31:39.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: osdmap e58: 9 total, 8 up, 9 in 2024-03-16T20:31:39.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: osdmap e59: 9 total, 8 up, 9 in 2024-03-16T20:31:39.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:39.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: from='osd.8 ' entity='osd.8' 2024-03-16T20:31:39.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:40.507 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-16T20:31:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] boot 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:31:40.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:31:40.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] boot 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:40.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: osd.8 [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] boot 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: osdmap e60: 9 total, 9 up, 9 in 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:31:40.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:31:42.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:41 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi189 2024-03-16T20:31:42.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:41 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-16T20:31:42.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:41 smithi023 ceph-mon[31879]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T20:31:42.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:41 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi189 2024-03-16T20:31:42.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:41 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-16T20:31:42.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:41 smithi031 ceph-mon[36281]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T20:31:42.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:41 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi189 2024-03-16T20:31:42.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:41 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-16T20:31:42.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:41 smithi189 ceph-mon[36655]: osdmap e61: 9 total, 9 up, 9 in 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 -- 172.21.15.23:0/3147393169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103bb0 msgr2=0x7fc8e010a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 --2- 172.21.15.23:0/3147393169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103bb0 0x7fc8e010a460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b0d0 tx=0x7fc8d800b3e0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 -- 172.21.15.23:0/3147393169 shutdown_connections 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 --2- 172.21.15.23:0/3147393169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103bb0 0x7fc8e010a460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 --2- 172.21.15.23:0/3147393169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 0x7fc8e0102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 --2- 172.21.15.23:0/3147393169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103210 0x7fc8e0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 -- 172.21.15.23:0/3147393169 >> 172.21.15.23:0/3147393169 conn(0x7fc8e00fd5f0 msgr2=0x7fc8e00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:31:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 -- 172.21.15.23:0/3147393169 shutdown_connections 2024-03-16T20:31:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.485+0000 7fc8e5209700 1 -- 172.21.15.23:0/3147393169 wait complete. 2024-03-16T20:31:42.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 Processor -- start 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 -- start start 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 0x7fc8e0075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103bb0 0x7fc8e0074910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0075090 con 0x7fc8e0103210 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0075200 con 0x7fc8e0103bb0 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8e5209700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e01a21a0 con 0x7fc8e0102010 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.486+0000 7fc8de59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36488/0 (socket says 172.21.15.23:36488) 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 -- 172.21.15.23:0/435105605 learned_addr learned my addr 172.21.15.23:0/435105605 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8ded9d700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 0x7fc8e0075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8df59e700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103bb0 0x7fc8e0074910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 -- 172.21.15.23:0/435105605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 msgr2=0x7fc8e0075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 0x7fc8e0075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 -- 172.21.15.23:0/435105605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103bb0 msgr2=0x7fc8e0074910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103bb0 0x7fc8e0074910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 -- 172.21.15.23:0/435105605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8009d00 con 0x7fc8e0103210 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8de59c700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c920 tx=0x7fc8d000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0007b00 con 0x7fc8e0103210 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8d000ce70 con 0x7fc8e0103210 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e01a2480 con 0x7fc8e0103210 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0005610 con 0x7fc8e0103210 2024-03-16T20:31:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.487+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e01a2a00 con 0x7fc8e0103210 2024-03-16T20:31:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.488+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e01093d0 con 0x7fc8e0103210 2024-03-16T20:31:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.489+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc8d0004750 con 0x7fc8e0103210 2024-03-16T20:31:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.490+0000 7fc8d7fff700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 0x7fc8cc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:31:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.490+0000 7fc8ded9d700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 0x7fc8cc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:31:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.490+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fc8d00cc9a0 con 0x7fc8e0103210 2024-03-16T20:31:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.490+0000 7fc8ded9d700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 0x7fc8cc0a3f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0103070 tx=0x7fc8c8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:31:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.495+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc8d0096540 con 0x7fc8e0103210 2024-03-16T20:31:42.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:31:42.710+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8e002d990 con 0x7fc8cc0a1ac0 2024-03-16T20:31:43.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:42 smithi023 ceph-mon[31879]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T20:31:43.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:42 smithi023 ceph-mon[31879]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:43.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:43.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:43.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:43.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:42 smithi031 ceph-mon[36281]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T20:31:43.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:42 smithi031 ceph-mon[36281]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:43.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:43.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:43.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:42 smithi189 ceph-mon[36655]: osdmap e62: 9 total, 9 up, 9 in 2024-03-16T20:31:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:42 smithi189 ceph-mon[36655]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:31:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:31:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:44.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:43 smithi031 ceph-mon[36281]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:44.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:43 smithi031 ceph-mon[36281]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T20:31:44.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:43 smithi189 ceph-mon[36655]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:44.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:43 smithi189 ceph-mon[36655]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T20:31:44.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:43 smithi023 ceph-mon[31879]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:31:44.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:43 smithi023 ceph-mon[31879]: osdmap e63: 9 total, 9 up, 9 in 2024-03-16T20:31:45.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:44 smithi023 ceph-mon[31879]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:45.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:44 smithi023 ceph-mon[31879]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T20:31:45.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:44 smithi031 ceph-mon[36281]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:45.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:44 smithi031 ceph-mon[36281]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T20:31:45.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:44 smithi189 ceph-mon[36655]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:45.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:44 smithi189 ceph-mon[36655]: osdmap e64: 9 total, 9 up, 9 in 2024-03-16T20:31:46.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:45 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]: dispatch 2024-03-16T20:31:46.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:45 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]': finished 2024-03-16T20:31:46.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:45 smithi031 ceph-mon[36281]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T20:31:46.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:31:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:45 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]: dispatch 2024-03-16T20:31:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:45 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]': finished 2024-03-16T20:31:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:45 smithi189 ceph-mon[36655]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T20:31:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:31:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:45 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]: dispatch 2024-03-16T20:31:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:45 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2312834040' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38b7a821-6c63-4636-84e2-cfa25c440244"}]': finished 2024-03-16T20:31:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:45 smithi023 ceph-mon[31879]: osdmap e65: 10 total, 9 up, 10 in 2024-03-16T20:31:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:31:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:46 smithi189 ceph-mon[36655]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:31:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:46 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/568073671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:47.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:46 smithi023 ceph-mon[31879]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:31:47.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:46 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/568073671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:47.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:46 smithi031 ceph-mon[36281]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:31:47.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:46 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/568073671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:31:49.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:48 smithi023 ceph-mon[31879]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:31:49.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:48 smithi189 ceph-mon[36655]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:31:49.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:48 smithi031 ceph-mon[36281]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:31:51.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:50 smithi023 ceph-mon[31879]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T20:31:51.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:50 smithi189 ceph-mon[36655]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T20:31:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:50 smithi031 ceph-mon[36281]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-16T20:31:52.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:52 smithi023 ceph-mon[31879]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:31:52.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:52 smithi031 ceph-mon[36281]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:31:52.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:52 smithi189 ceph-mon[36655]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:31:53.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T20:31:53.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:53.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T20:31:53.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:53.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-16T20:31:53.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:31:54.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:54 smithi023 ceph-mon[31879]: Deploying daemon osd.9 on smithi023 2024-03-16T20:31:54.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:54 smithi023 ceph-mon[31879]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T20:31:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:54 smithi031 ceph-mon[36281]: Deploying daemon osd.9 on smithi023 2024-03-16T20:31:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:54 smithi031 ceph-mon[36281]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T20:31:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:54 smithi189 ceph-mon[36655]: Deploying daemon osd.9 on smithi023 2024-03-16T20:31:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:54 smithi189 ceph-mon[36655]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T20:31:56.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:56 smithi023 ceph-mon[31879]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:56 smithi031 ceph-mon[36281]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:56.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:56 smithi189 ceph-mon[36655]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:58.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:31:58 smithi023 ceph-mon[31879]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:58.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:31:58 smithi031 ceph-mon[36281]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:31:58.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:31:58 smithi189 ceph-mon[36655]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:00.315 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.315 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.315 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:00.315 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:00 smithi023 ceph-mon[31879]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:00 smithi031 ceph-mon[36281]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:00 smithi189 ceph-mon[36655]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:02.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:02 smithi023 ceph-mon[31879]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:02.271 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:02 smithi189 ceph-mon[36655]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:02 smithi031 ceph-mon[36281]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:02.976 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2024-03-16T20:32:02.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.971+0000 7fc8d7fff700 1 -- 172.21.15.23:0/435105605 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc8e002d990 con 0x7fc8cc0a1ac0 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 msgr2=0x7fc8cc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 0x7fc8cc0a3f80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0103070 tx=0x7fc8c8009190 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 msgr2=0x7fc8e00743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c920 tx=0x7fc8d000cc30 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 shutdown_connections 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8cc0a1ac0 0x7fc8cc0a3f80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8e0103bb0 0x7fc8e0074910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8e0102010 0x7fc8e0075d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 --2- 172.21.15.23:0/435105605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0103210 0x7fc8e00743d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 >> 172.21.15.23:0/435105605 conn(0x7fc8e00fd5f0 msgr2=0x7fc8e0106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 shutdown_connections 2024-03-16T20:32:02.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:02.974+0000 7fc8e5209700 1 -- 172.21.15.23:0/435105605 wait complete. 2024-03-16T20:32:03.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:03 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:03 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:03.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.570 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:03.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:03 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:03.595 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.9.service 2024-03-16T20:32:03.598 INFO:tasks.cephadm:Deploying osd.10 on smithi031 with /dev/vg_nvme/lv_1... 2024-03-16T20:32:03.598 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T20:32:03.860 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:32:04.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-mon[31879]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T20:32:04.190 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-mon[31879]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T20:32:04.190 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-mon[31879]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:32:04.190 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:04.190 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-mon[31879]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:04.246 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:04 smithi031 ceph-mon[36281]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T20:32:04.247 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:04 smithi031 ceph-mon[36281]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T20:32:04.247 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:04 smithi031 ceph-mon[36281]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:32:04.247 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:04 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:04.247 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:04 smithi031 ceph-mon[36281]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:04 smithi189 ceph-mon[36655]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-16T20:32:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:04 smithi189 ceph-mon[36655]: osdmap e66: 10 total, 9 up, 10 in 2024-03-16T20:32:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:04 smithi189 ceph-mon[36655]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-16T20:32:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:04 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:04 smithi189 ceph-mon[36655]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:04.828 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:32:04.618+0000 7f761d7d5700 -1 osd.9 0 waiting for initial osdmap 2024-03-16T20:32:04.829 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:32:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:32:04.630+0000 7f76155c0700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:32:05.292 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:05 smithi031 ceph-mon[36281]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:32:05.292 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:05 smithi031 ceph-mon[36281]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T20:32:05.292 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.292 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:05 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.292 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:05 smithi031 ceph-mon[36281]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' 2024-03-16T20:32:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:05 smithi189 ceph-mon[36655]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:32:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:05 smithi189 ceph-mon[36655]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T20:32:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:05 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:05 smithi189 ceph-mon[36655]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' 2024-03-16T20:32:05.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:05 smithi023 ceph-mon[31879]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-16T20:32:05.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:05 smithi023 ceph-mon[31879]: osdmap e67: 10 total, 9 up, 10 in 2024-03-16T20:32:05.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:05 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:05.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:05 smithi023 ceph-mon[31879]: from='osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919]' entity='osd.9' 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110706 s, 947 MB/s 2024-03-16T20:32:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] boot 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T20:32:06.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:06 smithi023 ceph-mon[31879]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] boot 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.324 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:06 smithi031 ceph-mon[36281]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: osd.9 [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] boot 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: osdmap e68: 10 total, 10 up, 10 in 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-16T20:32:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:06 smithi189 ceph-mon[36655]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-16T20:32:06.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-03-16T20:32:06.932 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:32:07.934 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T20:32:07.934 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi023 2024-03-16T20:32:07.934 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.934 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.934 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:07.935 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:07 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi023 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:07.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:07 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: osdmap e69: 10 total, 10 up, 10 in 2024-03-16T20:32:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi023 2024-03-16T20:32:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:08.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:07 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 -- 172.21.15.31:0/342672665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102180 msgr2=0x7f133c1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 --2- 172.21.15.31:0/342672665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102180 0x7f133c1025e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f13300099c0 tx=0x7f1330009cd0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 -- 172.21.15.31:0/342672665 shutdown_connections 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 --2- 172.21.15.31:0/342672665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c102b20 0x7f133c1093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 --2- 172.21.15.31:0/342672665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c100f80 0x7f133c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 --2- 172.21.15.31:0/342672665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102180 0x7f133c1025e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 -- 172.21.15.31:0/342672665 >> 172.21.15.31:0/342672665 conn(0x7f133c0fc540 msgr2=0x7f133c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 -- 172.21.15.31:0/342672665 shutdown_connections 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.536+0000 7f1341233700 1 -- 172.21.15.31:0/342672665 wait complete. 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 Processor -- start 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 -- start start 2024-03-16T20:32:08.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c100f80 0x7f133c06b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c102180 0x7f133c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c19fae0 con 0x7f133c100f80 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c19fc50 con 0x7f133c102b20 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f1341233700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c19fe10 con 0x7f133c102180 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f133b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f133b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47448/0 (socket says 172.21.15.31:47448) 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f133b59e700 1 -- 172.21.15.31:0/4007808316 learned_addr learned my addr 172.21.15.31:0/4007808316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.537+0000 7f133a59c700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c102180 0x7f133c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133ad9d700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c100f80 0x7f133c06b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 -- 172.21.15.31:0/4007808316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c102180 msgr2=0x7f133c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c102180 0x7f133c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 -- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c100f80 msgr2=0x7f133c06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c100f80 0x7f133c06b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 -- 172.21.15.31:0/4007808316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300095f0 con 0x7f133c102b20 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f133b59e700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1334007ee0 tx=0x7f133400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133400ddd0 con 0x7f133c102b20 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f133400f040 con 0x7f133c102b20 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c1a00f0 con 0x7f133c102b20 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334015400 con 0x7f133c102b20 2024-03-16T20:32:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.538+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c1a0610 con 0x7f133c102b20 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.539+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c108410 con 0x7f133c102b20 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.540+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1334003a00 con 0x7f133c102b20 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.541+0000 7f1323fff700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 0x7f13240a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.541+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f13340cccf0 con 0x7f133c102b20 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.541+0000 7f133ad9d700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 0x7f13240a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.542+0000 7f133ad9d700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 0x7f13240a4000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f132c005fd0 tx=0x7f132c005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.545+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1334096660 con 0x7f133c102b20 2024-03-16T20:32:08.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:08.758+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f133c02d990 con 0x7f13240a1b40 2024-03-16T20:32:08.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:08 smithi031 ceph-mon[36281]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:08.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:08 smithi189 ceph-mon[36655]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:08 smithi023 ceph-mon[31879]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:09.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:09 smithi031 ceph-mon[36281]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:09.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:09 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:09.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:09 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:09.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:09 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:09 smithi189 ceph-mon[36655]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:09 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:09 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:09 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:10.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:09 smithi023 ceph-mon[31879]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:10.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:09 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:10.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:09 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:10.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:09 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:10.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:10 smithi031 ceph-mon[36281]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:10.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:10 smithi189 ceph-mon[36655]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:11.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:10 smithi023 ceph-mon[31879]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:11.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:11 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/3620101664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:11.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:11 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:11.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:11 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]': finished 2024-03-16T20:32:11.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:11 smithi031 ceph-mon[36281]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T20:32:11.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:11.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:11 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/3620101664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:11.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:11 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:11.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:11 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]': finished 2024-03-16T20:32:11.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:11 smithi189 ceph-mon[36655]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T20:32:11.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:11 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/3620101664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:11 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]: dispatch 2024-03-16T20:32:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:11 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6615be9c-9a48-448d-af63-51c8ab219057"}]': finished 2024-03-16T20:32:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:11 smithi023 ceph-mon[31879]: osdmap e70: 11 total, 10 up, 11 in 2024-03-16T20:32:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:12.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:12 smithi031 ceph-mon[36281]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:12.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:12 smithi031 ceph-mon[36281]: from='client.? 172.21.15.31:0/103149479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:12.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:12 smithi189 ceph-mon[36655]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:12.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:12 smithi189 ceph-mon[36655]: from='client.? 172.21.15.31:0/103149479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:13.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:12 smithi023 ceph-mon[31879]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:13.133 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:12 smithi023 ceph-mon[31879]: from='client.? 172.21.15.31:0/103149479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:14.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:14 smithi023 ceph-mon[31879]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:14 smithi031 ceph-mon[36281]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:14 smithi189 ceph-mon[36655]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:16.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:16 smithi023 ceph-mon[31879]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:16.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:16 smithi031 ceph-mon[36281]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:16.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:16 smithi189 ceph-mon[36655]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:18.331 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:18 smithi031 ceph-mon[36281]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:18 smithi023 ceph-mon[31879]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:18.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:18 smithi189 ceph-mon[36655]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:19.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T20:32:19.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:19 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:19.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T20:32:19.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:19 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:19.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-16T20:32:19.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:19 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:20 smithi031 ceph-mon[36281]: Deploying daemon osd.10 on smithi031 2024-03-16T20:32:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:20 smithi031 ceph-mon[36281]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:20.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:20 smithi023 ceph-mon[31879]: Deploying daemon osd.10 on smithi031 2024-03-16T20:32:20.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:20 smithi023 ceph-mon[31879]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:20.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:20 smithi189 ceph-mon[36655]: Deploying daemon osd.10 on smithi031 2024-03-16T20:32:20.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:20 smithi189 ceph-mon[36655]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:22.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:22 smithi031 ceph-mon[36281]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:22.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:22 smithi023 ceph-mon[31879]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:22.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:22 smithi189 ceph-mon[36655]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:24.293 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:24 smithi031 ceph-mon[36281]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:24.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:24 smithi023 ceph-mon[31879]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:24 smithi189 ceph-mon[36655]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:26.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:26.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:26.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:26.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:27.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:26 smithi031 ceph-mon[36281]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:27.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:26 smithi189 ceph-mon[36655]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:27.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:26 smithi023 ceph-mon[31879]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:28.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.729+0000 7f1323fff700 1 -- 172.21.15.31:0/4007808316 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f133c02d990 con 0x7f13240a1b40 2024-03-16T20:32:28.734 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 10 on host 'smithi031' 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.731+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 msgr2=0x7f13240a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.731+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 0x7f13240a4000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f132c005fd0 tx=0x7f132c005dd0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.731+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 msgr2=0x7f133c19f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.731+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1334007ee0 tx=0x7f133400d3b0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 shutdown_connections 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13240a1b40 0x7f13240a4000 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f133c100f80 0x7f133c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f133c102b20 0x7f133c19f340 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 --2- 172.21.15.31:0/4007808316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f133c102180 0x7f133c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:28.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 >> 172.21.15.31:0/4007808316 conn(0x7f133c0fc540 msgr2=0x7f133c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:28.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 shutdown_connections 2024-03-16T20:32:28.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-03-16T20:32:28.732+0000 7f1341233700 1 -- 172.21.15.31:0/4007808316 wait complete. 2024-03-16T20:32:28.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:28.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:28.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:28 smithi031 ceph-mon[36281]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:29.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.142 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:29.143 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:28 smithi023 ceph-mon[31879]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:29.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:29.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:29.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:28 smithi189 ceph-mon[36655]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-16T20:32:29.433 DEBUG:teuthology.orchestra.run.smithi031:osd.10> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.10.service 2024-03-16T20:32:29.436 INFO:tasks.cephadm:Deploying osd.11 on smithi189 with /dev/vg_nvme/lv_1... 2024-03-16T20:32:29.436 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-16T20:32:29.683 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:32:30.804 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T20:32:30.804 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T20:32:30.805 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:30.805 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:30.805 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:30.805 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-mon[36281]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:30.805 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:32:30.730+0000 7f11c5661700 -1 osd.10 0 waiting for initial osdmap 2024-03-16T20:32:30.805 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:32:30 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:32:30.743+0000 7f11c1454700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:32:31.088 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T20:32:31.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T20:32:31.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:31.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:31.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:31.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:30 smithi189 ceph-mon[36655]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-16T20:32:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: osdmap e71: 11 total, 10 up, 11 in 2024-03-16T20:32:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: from='osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-03-16T20:32:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:30 smithi023 ceph-mon[31879]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113697 s, 922 MB/s 2024-03-16T20:32:31.813 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-16T20:32:31.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:31 smithi031 ceph-mon[36281]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:32:31.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:31 smithi031 ceph-mon[36281]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T20:32:31.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:31 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:31.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:31 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:31 smithi023 ceph-mon[31879]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:32:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:31 smithi023 ceph-mon[31879]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T20:32:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:31 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:31 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:31 smithi189 ceph-mon[36655]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-03-16T20:32:32.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:31 smithi189 ceph-mon[36655]: osdmap e72: 11 total, 10 up, 11 in 2024-03-16T20:32:32.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:31 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.143 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:31 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-03-16T20:32:32.934 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] boot 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:32.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:32.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:32 smithi189 ceph-mon[36655]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T20:32:33.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] boot 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:32 smithi023 ceph-mon[31879]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: osd.10 [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] boot 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: osdmap e73: 11 total, 11 up, 11 in 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:33.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:32 smithi031 ceph-mon[36281]: osdmap e74: 11 total, 11 up, 11 in 2024-03-16T20:32:33.770 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:33 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi031 2024-03-16T20:32:33.770 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:33 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi031 to 4271M 2024-03-16T20:32:33.770 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:33 smithi031 ceph-mon[36281]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T20:32:34.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:33 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi031 2024-03-16T20:32:34.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:33 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi031 to 4271M 2024-03-16T20:32:34.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:33 smithi189 ceph-mon[36655]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T20:32:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:33 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi031 2024-03-16T20:32:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:33 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi031 to 4271M 2024-03-16T20:32:34.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:33 smithi023 ceph-mon[31879]: osdmap e75: 11 total, 11 up, 11 in 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- 172.21.15.189:0/1549092473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c01021b0 msgr2=0x7f99c0102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- 172.21.15.189:0/1549092473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c01021b0 0x7f99c0102610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009990 tx=0x7f99b0009ca0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- 172.21.15.189:0/1549092473 shutdown_connections 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- 172.21.15.189:0/1549092473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 0x7f99c0109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- 172.21.15.189:0/1549092473 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c0100fb0 0x7f99c01013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- 172.21.15.189:0/1549092473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c01021b0 0x7f99c0102610 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- 172.21.15.189:0/1549092473 >> 172.21.15.189:0/1549092473 conn(0x7f99c00fc570 msgr2=0x7f99c00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- 172.21.15.189:0/1549092473 shutdown_connections 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- 172.21.15.189:0/1549092473 wait complete. 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 Processor -- start 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 -- start start 2024-03-16T20:32:34.713 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.706+0000 7f99c729a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c01021b0 0x7f99c0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 0x7f99c019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019fb60 con 0x7f99c0102b50 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019fca0 con 0x7f99c01021b0 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c019fe30 con 0x7f99c0100fb0 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38214/0 (socket says 172.21.15.189:38214) 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 -- 172.21.15.189:0/1320729486 learned_addr learned my addr 172.21.15.189:0/1320729486 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c4835700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c01021b0 0x7f99c0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 -- 172.21.15.189:0/1320729486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c01021b0 msgr2=0x7f99c0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c01021b0 0x7f99c0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 -- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 msgr2=0x7f99c019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5837700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 0x7f99c019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 0x7f99c019f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 -- 172.21.15.189:0/1320729486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00095f0 con 0x7f99c0100fb0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c5036700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00e9a0 tx=0x7f99bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc009790 con 0x7f99c0100fb0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c01a0110 con 0x7f99c0100fb0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99bc004500 con 0x7f99c0100fb0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc003e30 con 0x7f99c0100fb0 2024-03-16T20:32:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.707+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c01a0610 con 0x7f99c0100fb0 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.708+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c0068ca0 con 0x7f99c0100fb0 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.709+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f99bc003f90 con 0x7f99c0100fb0 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.710+0000 7f99b67fc700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 0x7f99ac0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.710+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f99bc014070 con 0x7f99c0100fb0 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.710+0000 7f99c4835700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 0x7f99ac0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.711+0000 7f99c4835700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 0x7f99ac0a3f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f99b000b720 tx=0x7f99b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:34.719 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.715+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f99bc0968e0 con 0x7f99c0100fb0 2024-03-16T20:32:34.924 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:34.919+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c006adf0 con 0x7f99ac0a1ac0 2024-03-16T20:32:34.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:34 smithi189 ceph-mon[36655]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T20:32:34.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:34 smithi189 ceph-mon[36655]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:35.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:34 smithi031 ceph-mon[36281]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T20:32:35.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:34 smithi031 ceph-mon[36281]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:35.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:34 smithi023 ceph-mon[31879]: osdmap e76: 11 total, 11 up, 11 in 2024-03-16T20:32:35.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:34 smithi023 ceph-mon[31879]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:36.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:35 smithi031 ceph-mon[36281]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T20:32:36.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:35 smithi031 ceph-mon[36281]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:36.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:35 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:36.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:35 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:36.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:35 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:35 smithi189 ceph-mon[36655]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T20:32:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:35 smithi189 ceph-mon[36655]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:35 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:35 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:35 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:36.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:35 smithi023 ceph-mon[31879]: osdmap e77: 11 total, 11 up, 11 in 2024-03-16T20:32:36.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:35 smithi023 ceph-mon[31879]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:32:36.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:35 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T20:32:36.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:35 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T20:32:36.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:35 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:37.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:36 smithi031 ceph-mon[36281]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:32:37.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:36 smithi189 ceph-mon[36655]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:32:37.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:36 smithi023 ceph-mon[31879]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T20:32:38.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:37 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/3314945418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:37 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:37 smithi031 ceph-mon[36281]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]': finished 2024-03-16T20:32:38.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:37 smithi031 ceph-mon[36281]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T20:32:38.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:38.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:37 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/3314945418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:37 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:37 smithi189 ceph-mon[36655]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]': finished 2024-03-16T20:32:38.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:37 smithi189 ceph-mon[36655]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T20:32:38.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:37 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/3314945418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:37 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]: dispatch 2024-03-16T20:32:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:37 smithi023 ceph-mon[31879]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7877d354-a2b5-4c0e-844d-b56717ad5a9b"}]': finished 2024-03-16T20:32:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:37 smithi023 ceph-mon[31879]: osdmap e78: 12 total, 11 up, 12 in 2024-03-16T20:32:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:39.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:38 smithi031 ceph-mon[36281]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-16T20:32:39.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:38 smithi031 ceph-mon[36281]: from='client.? 172.21.15.189:0/3930272407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:39.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:38 smithi189 ceph-mon[36655]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-16T20:32:39.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:38 smithi189 ceph-mon[36655]: from='client.? 172.21.15.189:0/3930272407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:39.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:38 smithi023 ceph-mon[31879]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-16T20:32:39.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:38 smithi023 ceph-mon[31879]: from='client.? 172.21.15.189:0/3930272407' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T20:32:41.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:40 smithi189 ceph-mon[36655]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:32:41.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:40 smithi023 ceph-mon[31879]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:32:41.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:40 smithi031 ceph-mon[36281]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T20:32:43.114 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:42 smithi189 ceph-mon[36655]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:32:43.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:42 smithi023 ceph-mon[31879]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:32:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:42 smithi031 ceph-mon[36281]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T20:32:45.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:44 smithi031 ceph-mon[36281]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:45.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:44 smithi189 ceph-mon[36655]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:45.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:44 smithi023 ceph-mon[31879]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T20:32:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T20:32:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-16T20:32:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:32:47.186 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:46 smithi189 ceph-mon[36655]: Deploying daemon osd.11 on smithi189 2024-03-16T20:32:47.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:46 smithi189 ceph-mon[36655]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:47.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:46 smithi023 ceph-mon[31879]: Deploying daemon osd.11 on smithi189 2024-03-16T20:32:47.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:46 smithi023 ceph-mon[31879]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:47.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:46 smithi031 ceph-mon[36281]: Deploying daemon osd.11 on smithi189 2024-03-16T20:32:47.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:46 smithi031 ceph-mon[36281]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:48.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:47 smithi189 ceph-mon[36655]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:48.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:47 smithi023 ceph-mon[31879]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:48.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:47 smithi031 ceph-mon[36281]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:50.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:50 smithi023 ceph-mon[31879]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:50.384 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:50 smithi189 ceph-mon[36655]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:50.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:50 smithi031 ceph-mon[36281]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:52.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:52.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:51 smithi023 ceph-mon[31879]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:52.354 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.354 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.354 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:52.354 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:51 smithi189 ceph-mon[36655]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:52.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.441 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:52.441 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:32:52.441 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:51 smithi031 ceph-mon[36281]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:54.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:54 smithi189 ceph-mon[36655]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:54.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:54 smithi023 ceph-mon[31879]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:54 smithi031 ceph-mon[36281]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:54.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.736+0000 7f99b67fc700 1 -- 172.21.15.189:0/1320729486 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f99c006adf0 con 0x7f99ac0a1ac0 2024-03-16T20:32:54.743 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 11 on host 'smithi189' 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 msgr2=0x7f99ac0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 0x7f99ac0a3f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f99b000b720 tx=0x7f99b0005fd0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 msgr2=0x7f99c0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00e9a0 tx=0x7f99bc00ecb0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 shutdown_connections 2024-03-16T20:32:54.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99ac0a1ac0 0x7f99ac0a3f80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c0102b50 0x7f99c019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99c0100fb0 0x7f99c0197fd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 --2- 172.21.15.189:0/1320729486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99c01021b0 0x7f99c0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 >> 172.21.15.189:0/1320729486 conn(0x7f99c00fc570 msgr2=0x7f99c006a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 shutdown_connections 2024-03-16T20:32:54.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:32:54.738+0000 7f99c729a700 1 -- 172.21.15.189:0/1320729486 wait complete. 2024-03-16T20:32:55.384 DEBUG:teuthology.orchestra.run.smithi189:osd.11> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.11.service 2024-03-16T20:32:55.386 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-16T20:32:55.387 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T20:32:55.640 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:32:55.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:55.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:55.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:55.990 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:55.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:55.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:56.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-16T20:32:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:32:56.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T20:32:56.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T20:32:56.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:56.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:56.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:56.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:56 smithi023 ceph-mon[31879]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:56.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:56 smithi189 ceph-mon[36655]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: osdmap e79: 12 total, 11 up, 12 in 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: from='osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-16T20:32:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:56 smithi031 ceph-mon[36281]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:57.416 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:32:57.105+0000 7f222b42d700 -1 osd.11 0 waiting for initial osdmap 2024-03-16T20:32:57.417 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:32:57.123+0000 7f2223218700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T20:32:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- 172.21.15.23:0/2972823881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb00749c0 msgr2=0x7febb0074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2972823881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb00749c0 0x7febb0074e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feba4009990 tx=0x7feba4009ca0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- 172.21.15.23:0/2972823881 shutdown_connections 2024-03-16T20:32:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2972823881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0109f10 0x7febb010c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2972823881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb0075f90 0x7febb00743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2972823881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb00749c0 0x7febb0074e40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- 172.21.15.23:0/2972823881 >> 172.21.15.23:0/2972823881 conn(0x7febb00fd5f0 msgr2=0x7febb00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- 172.21.15.23:0/2972823881 shutdown_connections 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- 172.21.15.23:0/2972823881 wait complete. 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 Processor -- start 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.498+0000 7febb5ee9700 1 -- start start 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb00749c0 0x7febb019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0075f90 0x7febb019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb01a1d20 con 0x7febb0109f10 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb01a1e60 con 0x7febb0075f90 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febb5ee9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb01a1ff0 con 0x7febb00749c0 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58718/0 (socket says 172.21.15.23:58718) 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 -- 172.21.15.23:0/2006935098 learned_addr learned my addr 172.21.15.23:0/2006935098 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaeffd700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0075f90 0x7febb019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaf7fe700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb00749c0 0x7febb019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 -- 172.21.15.23:0/2006935098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb00749c0 msgr2=0x7febb019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb00749c0 0x7febb019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 -- 172.21.15.23:0/2006935098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0075f90 msgr2=0x7febb019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0075f90 0x7febb019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 -- 172.21.15.23:0/2006935098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40095f0 con 0x7febb0109f10 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.499+0000 7febaffff700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feba8007f10 tx=0x7feba800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.500+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8017070 con 0x7febb0109f10 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.500+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feba800f040 con 0x7febb0109f10 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.500+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb01a22d0 con 0x7febb0109f10 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.500+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8005650 con 0x7febb0109f10 2024-03-16T20:32:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.500+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb01a2780 con 0x7febb0109f10 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.501+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb0108d20 con 0x7febb0109f10 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.502+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7feba80057b0 con 0x7febb0109f10 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.502+0000 7febacff9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 0x7feb980a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.503+0000 7febaf7fe700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 0x7feb980a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.503+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6847+0+0 (secure 0 0 0) 0x7feba80cccb0 con 0x7febb0109f10 2024-03-16T20:32:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.503+0000 7febaf7fe700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 0x7feb980a3f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feba0005950 tx=0x7feba000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:32:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.506+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7feba8096480 con 0x7febb0109f10 2024-03-16T20:32:57.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.711+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7febb010fb40 con 0x7febb0109f10 2024-03-16T20:32:57.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.712+0000 7febacff9700 1 -- 172.21.15.23:0/2006935098 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7feba8099b40 con 0x7febb0109f10 2024-03-16T20:32:57.717 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.713+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 msgr2=0x7feb980a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 0x7feb980a3f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feba0005950 tx=0x7feba000a300 comp rx=0 tx=0).stop 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 msgr2=0x7febb01a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feba8007f10 tx=0x7feba800d3b0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 shutdown_connections 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feb980a1ac0 0x7feb980a3f80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febb0109f10 0x7febb01a1750 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb00749c0 0x7febb019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 --2- 172.21.15.23:0/2006935098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febb0075f90 0x7febb019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:32:57.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 >> 172.21.15.23:0/2006935098 conn(0x7febb00fd5f0 msgr2=0x7febb0105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:32:57.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 shutdown_connections 2024-03-16T20:32:57.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:32:57.714+0000 7febb5ee9700 1 -- 172.21.15.23:0/2006935098 wait complete. 2024-03-16T20:32:57.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-mon[36655]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:32:57.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-mon[36655]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T20:32:57.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:57.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-mon[36655]: from='osd.11 ' entity='osd.11' 2024-03-16T20:32:57.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:57 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:57 smithi023 ceph-mon[31879]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:32:58.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:57 smithi023 ceph-mon[31879]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T20:32:58.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:57 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:57 smithi023 ceph-mon[31879]: from='osd.11 ' entity='osd.11' 2024-03-16T20:32:58.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:57 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:57 smithi031 ceph-mon[36281]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-16T20:32:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:57 smithi031 ceph-mon[36281]: osdmap e80: 12 total, 11 up, 12 in 2024-03-16T20:32:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:57 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:57 smithi031 ceph-mon[36281]: from='osd.11 ' entity='osd.11' 2024-03-16T20:32:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:57 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.512 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1710621151,"num_in_osds":12,"osd_in_since":1710621157,"num_remapped_pgs":0} 2024-03-16T20:32:58.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: purged_snaps scrub starts 2024-03-16T20:32:58.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: purged_snaps scrub ok 2024-03-16T20:32:58.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2006935098' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:32:58.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:58.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] boot 2024-03-16T20:32:58.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T20:32:58.787 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:58 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:58.922 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: purged_snaps scrub starts 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: purged_snaps scrub ok 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2006935098' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] boot 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T20:32:58.923 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:58 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: purged_snaps scrub starts 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: purged_snaps scrub ok 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2006935098' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: osd.11 [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] boot 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: osdmap e81: 12 total, 12 up, 12 in 2024-03-16T20:32:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:58 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-16T20:32:59.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T20:32:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:33:00.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: Detected new or changed devices on smithi189 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T20:33:00.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:32:59 smithi023 ceph-mon[31879]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: Detected new or changed devices on smithi189 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T20:33:00.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:32:59 smithi031 ceph-mon[36281]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: Detected new or changed devices on smithi189 2024-03-16T20:33:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:00.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: osdmap e82: 12 total, 12 up, 12 in 2024-03-16T20:33:00.473 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:32:59 smithi189 ceph-mon[36655]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:01.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 -- 172.21.15.23:0/2995755740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 msgr2=0x7f56a0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 --2- 172.21.15.23:0/2995755740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a0101330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5690009990 tx=0x7f5690009ca0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 -- 172.21.15.23:0/2995755740 shutdown_connections 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 --2- 172.21.15.23:0/2995755740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a0101330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 --2- 172.21.15.23:0/2995755740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102a30 0x7f56a0109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 --2- 172.21.15.23:0/2995755740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102070 0x7f56a01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 -- 172.21.15.23:0/2995755740 >> 172.21.15.23:0/2995755740 conn(0x7f56a00fc4b0 msgr2=0x7f56a00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 -- 172.21.15.23:0/2995755740 shutdown_connections 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.501+0000 7f56a62bb700 1 -- 172.21.15.23:0/2995755740 wait complete. 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 Processor -- start 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 -- start start 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 0x7f56a019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102a30 0x7f56a019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019fb30 con 0x7f56a0100f10 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019fca0 con 0x7f56a0102a30 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f56a62bb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019fe60 con 0x7f56a0102070 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f569ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f569ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55996/0 (socket says 172.21.15.23:55996) 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.502+0000 7f569ffff700 1 -- 172.21.15.23:0/3498486124 learned_addr learned my addr 172.21.15.23:0/3498486124 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569f7fe700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 0x7f56a019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f56a4858700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102a30 0x7f56a019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 -- 172.21.15.23:0/3498486124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 msgr2=0x7f56a019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 0x7f56a019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 -- 172.21.15.23:0/3498486124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102a30 msgr2=0x7f56a019f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102a30 0x7f56a019f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 -- 172.21.15.23:0/3498486124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900095f0 con 0x7f56a0100f10 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569f7fe700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 0x7f56a019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569ffff700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56900059d0 tx=0x7f5690005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569001d070 con 0x7f56a0100f10 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a01a00e0 con 0x7f56a0100f10 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5690004170 con 0x7f56a0100f10 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569000f460 con 0x7f56a0100f10 2024-03-16T20:33:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.503+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a01a0500 con 0x7f56a0100f10 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.504+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a01082c0 con 0x7f56a0100f10 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.505+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5690022590 con 0x7f56a0100f10 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.506+0000 7f569d7fa700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 0x7f56880a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.506+0000 7f569f7fe700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 0x7f56880a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.506+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56900ce8d0 con 0x7f56a0100f10 2024-03-16T20:33:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.506+0000 7f569f7fe700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 0x7f56880a3f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5694005950 tx=0x7f56940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:01.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.509+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5690098010 con 0x7f56a0100f10 2024-03-16T20:33:01.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.713+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f56a010c860 con 0x7f56a0100f10 2024-03-16T20:33:01.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.713+0000 7f569d7fa700 1 -- 172.21.15.23:0/3498486124 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7f569009b6d0 con 0x7f56a0100f10 2024-03-16T20:33:01.719 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 msgr2=0x7f56880a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 0x7f56880a3f80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5694005950 tx=0x7f56940058e0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 msgr2=0x7f56a006a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56900059d0 tx=0x7f5690005ef0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 shutdown_connections 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f56880a1ac0 0x7f56880a3f80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0100f10 0x7f56a006a850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56a0102a30 0x7f56a019f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 --2- 172.21.15.23:0/3498486124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a0102070 0x7f56a019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.715+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 >> 172.21.15.23:0/3498486124 conn(0x7f56a00fc4b0 msgr2=0x7f56a0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.716+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 shutdown_connections 2024-03-16T20:33:01.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:01.716+0000 7f56a62bb700 1 -- 172.21.15.23:0/3498486124 wait complete. 2024-03-16T20:33:01.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3498486124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:33:02.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3498486124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:33:02.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3498486124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T20:33:02.453 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1710621178,"num_in_osds":12,"osd_in_since":1710621157,"num_remapped_pgs":0} 2024-03-16T20:33:02.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T20:33:02.695 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:33:02.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:02 smithi189 ceph-mon[36655]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:03.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:02 smithi023 ceph-mon[31879]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:03.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:02 smithi031 ceph-mon[36281]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:04.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:04 smithi023 ceph-mon[31879]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:04.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2726034374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c1061c0 msgr2=0x7f5f6c10dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:04.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2726034374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c10dd90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b0a0 tx=0x7f5f6400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2726034374 shutdown_connections 2024-03-16T20:33:04.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2726034374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c10dd90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2726034374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c101170 0x7f5f6c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2726034374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2726034374 >> 172.21.15.23:0/2726034374 conn(0x7f5f6c076f30 msgr2=0x7f5f6c077340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2726034374 shutdown_connections 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.364+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2726034374 wait complete. 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 Processor -- start 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 -- start start 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c19bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c19c3d0 con 0x7f5f6c101170 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c19c540 con 0x7f5f6c1061c0 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f73ff3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c19c700 con 0x7f5f6c101b20 2024-03-16T20:33:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56026/0 (socket says 172.21.15.23:56026) 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 -- 172.21.15.23:0/2559303268 learned_addr learned my addr 172.21.15.23:0/2559303268 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f72590700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c19bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f7158e700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 -- 172.21.15.23:0/2559303268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 msgr2=0x7f5f6c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 -- 172.21.15.23:0/2559303268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c1061c0 msgr2=0x7f5f6c19bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c19bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f71d8f700 1 -- 172.21.15.23:0/2559303268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64009d00 con 0x7f5f6c101170 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.365+0000 7f5f7158e700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f71d8f700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000d950 tx=0x7f5f6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60009940 con 0x7f5f6c101170 2024-03-16T20:33:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c19c9e0 con 0x7f5f6c101170 2024-03-16T20:33:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f60010460 con 0x7f5f6c101170 2024-03-16T20:33:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000f620 con 0x7f5f6c101170 2024-03-16T20:33:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.366+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1a4240 con 0x7f5f6c101170 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.367+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c1094e0 con 0x7f5f6c101170 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.368+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5f600105d0 con 0x7f5f6c101170 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.368+0000 7f5f5effd700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 0x7f5f580a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.369+0000 7f5f7158e700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 0x7f5f580a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.369+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5f600cce00 con 0x7f5f6c101170 2024-03-16T20:33:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.369+0000 7f5f7158e700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 0x7f5f580a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005e90 tx=0x7f5f68005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:04.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.372+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5f60099fe0 con 0x7f5f6c101170 2024-03-16T20:33:04.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:04 smithi031 ceph-mon[36281]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:04 smithi189 ceph-mon[36655]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.576+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f6c068ca0 con 0x7f5f6c101170 2024-03-16T20:33:04.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.577+0000 7f5f5effd700 1 -- 172.21.15.23:0/2559303268 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f5f60099b70 con 0x7f5f6c101170 2024-03-16T20:33:04.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:04.583 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:32:59.716148+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 msgr2=0x7f5f580a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 0x7f5f580a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005e90 tx=0x7f5f68005d20 comp rx=0 tx=0).stop 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 msgr2=0x7f5f6c1974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000d950 tx=0x7f5f6000dc60 comp rx=0 tx=0).stop 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 shutdown_connections 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5f580a1ac0 0x7f5f580a3f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f6c101170 0x7f5f6c1974f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f6c1061c0 0x7f5f6c19bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 --2- 172.21.15.23:0/2559303268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c101b20 0x7f5f6c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 >> 172.21.15.23:0/2559303268 conn(0x7f5f6c076f30 msgr2=0x7f5f6c0fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 shutdown_connections 2024-03-16T20:33:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:04.579+0000 7f5f73ff3700 1 -- 172.21.15.23:0/2559303268 wait complete. 2024-03-16T20:33:05.193 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2559303268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:05.222 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T20:33:05.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T20:33:05.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2559303268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:33:05.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2559303268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:06 smithi023 ceph-mon[31879]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:06 smithi031 ceph-mon[36281]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:06 smithi189 ceph-mon[36655]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:07.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 -- 172.21.15.23:0/3089760094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674108010 msgr2=0x7f76741083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:07.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 --2- 172.21.15.23:0/3089760094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674108010 0x7f76741083f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f76680099c0 tx=0x7f7668009cd0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 -- 172.21.15.23:0/3089760094 shutdown_connections 2024-03-16T20:33:07.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 --2- 172.21.15.23:0/3089760094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674108010 0x7f76741083f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 --2- 172.21.15.23:0/3089760094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76741029f0 0x7f767410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 --2- 172.21.15.23:0/3089760094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7674102030 0x7f76741024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 -- 172.21.15.23:0/3089760094 >> 172.21.15.23:0/3089760094 conn(0x7f7674079110 msgr2=0x7f7674079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 -- 172.21.15.23:0/3089760094 shutdown_connections 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.209+0000 7f767b12d700 1 -- 172.21.15.23:0/3089760094 wait complete. 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 Processor -- start 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 -- start start 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76741029f0 0x7f7674074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7674108010 0x7f7674074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76741a68a0 con 0x7f7674102030 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76741a6a10 con 0x7f76741029f0 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f767b12d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76741a6bd0 con 0x7f7674108010 2024-03-16T20:33:07.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56042/0 (socket says 172.21.15.23:56042) 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 -- 172.21.15.23:0/2716485826 learned_addr learned my addr 172.21.15.23:0/2716485826 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f76796ca700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7674108010 0x7f7674074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7673fff700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76741029f0 0x7f7674074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 -- 172.21.15.23:0/2716485826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7674108010 msgr2=0x7f7674074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7674108010 0x7f7674074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 -- 172.21.15.23:0/2716485826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76741029f0 msgr2=0x7f7674074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76741029f0 0x7f7674074820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.210+0000 7f7678ec9700 1 -- 172.21.15.23:0/2716485826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680095f0 con 0x7f7674102030 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f7678ec9700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76680098e0 tx=0x7f7668005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766801d070 con 0x7f7674102030 2024-03-16T20:33:07.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7668003f50 con 0x7f7674102030 2024-03-16T20:33:07.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76741a6e50 con 0x7f7674102030 2024-03-16T20:33:07.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668022d20 con 0x7f7674102030 2024-03-16T20:33:07.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.211+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76741a7260 con 0x7f7674102030 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.212+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767410a4f0 con 0x7f7674102030 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.213+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7668022470 con 0x7f7674102030 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.214+0000 7f7671ffb700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 0x7f76640a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.214+0000 7f7673fff700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 0x7f76640a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.214+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76680ce3c0 con 0x7f7674102030 2024-03-16T20:33:07.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.214+0000 7f7673fff700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 0x7f76640a4000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7674079780 tx=0x7f766000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:07.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.218+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7668097a60 con 0x7f7674102030 2024-03-16T20:33:07.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.422+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7674112010 con 0x7f7674102030 2024-03-16T20:33:07.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.423+0000 7f7671ffb700 1 -- 172.21.15.23:0/2716485826 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f766809b120 con 0x7f7674102030 2024-03-16T20:33:07.428 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 msgr2=0x7f76640a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 0x7f76640a4000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7674079780 tx=0x7f766000a300 comp rx=0 tx=0).stop 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 msgr2=0x7f76740742e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76680098e0 tx=0x7f7668005740 comp rx=0 tx=0).stop 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 shutdown_connections 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f76640a1b40 0x7f76640a4000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674102030 0x7f76740742e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7674108010 0x7f7674074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 --2- 172.21.15.23:0/2716485826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76741029f0 0x7f7674074820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 >> 172.21.15.23:0/2716485826 conn(0x7f7674079110 msgr2=0x7f7674100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 shutdown_connections 2024-03-16T20:33:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:07.425+0000 7f767b12d700 1 -- 172.21.15.23:0/2716485826 wait complete. 2024-03-16T20:33:07.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:07 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2716485826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:07.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:07 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2716485826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:07 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2716485826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:08.172 INFO:tasks.cephadm:Adding mds.a on smithi023 2024-03-16T20:33:08.172 INFO:tasks.cephadm:Adding mds.d on smithi023 2024-03-16T20:33:08.172 INFO:tasks.cephadm:Adding mds.g on smithi023 2024-03-16T20:33:08.172 INFO:tasks.cephadm:Adding mds.j on smithi023 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.b on smithi031 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.e on smithi031 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.h on smithi031 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.k on smithi031 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.c on smithi189 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.f on smithi189 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.i on smithi189 2024-03-16T20:33:08.173 INFO:tasks.cephadm:Adding mds.l on smithi189 2024-03-16T20:33:08.173 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l' 2024-03-16T20:33:08.428 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:33:08.792 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:08 smithi189 ceph-mon[36655]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:08.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:08 smithi023 ceph-mon[31879]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:08.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:08 smithi031 ceph-mon[36281]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 -- 172.21.15.189:0/2382464762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 msgr2=0x7f8370102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 --2- 172.21.15.189:0/2382464762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f8370102430 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8360009990 tx=0x7f8360009ca0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 -- 172.21.15.189:0/2382464762 shutdown_connections 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 --2- 172.21.15.189:0/2382464762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103bb0 0x7f837010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 --2- 172.21.15.189:0/2382464762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f8370102430 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 --2- 172.21.15.189:0/2382464762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103210 0x7f8370103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 -- 172.21.15.189:0/2382464762 >> 172.21.15.189:0/2382464762 conn(0x7f83700fd5f0 msgr2=0x7f83700ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 -- 172.21.15.189:0/2382464762 shutdown_connections 2024-03-16T20:33:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.168+0000 7f837679f700 1 -- 172.21.15.189:0/2382464762 wait complete. 2024-03-16T20:33:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 Processor -- start 2024-03-16T20:33:10.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 -- start start 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 0x7f837019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103bb0 0x7f83701a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83701a1da0 con 0x7f8370103210 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83701a1ee0 con 0x7f8370103bb0 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f837679f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83701a2070 con 0x7f8370102010 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f836ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f836ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57622/0 (socket says 172.21.15.189:57622) 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.169+0000 7f836ffff700 1 -- 172.21.15.189:0/1139988793 learned_addr learned my addr 172.21.15.189:0/1139988793 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f8374d3c700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103bb0 0x7f83701a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836f7fe700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 0x7f837019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:10.177 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 -- 172.21.15.189:0/1139988793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103bb0 msgr2=0x7f83701a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103bb0 0x7f83701a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 -- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 msgr2=0x7f837019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 0x7f837019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 -- 172.21.15.189:0/1139988793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600095f0 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836f7fe700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 0x7f837019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836ffff700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8360003930 tx=0x7f836000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836001d070 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f836000fb40 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83701a22f0 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83600056a0 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.170+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83701a2760 con 0x7f8370102010 2024-03-16T20:33:10.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.171+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8370068ca0 con 0x7f8370102010 2024-03-16T20:33:10.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.172+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f836000fcb0 con 0x7f8370102010 2024-03-16T20:33:10.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.173+0000 7f836d7fa700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 0x7f83580a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:10.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.173+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f83600ce100 con 0x7f8370102010 2024-03-16T20:33:10.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.173+0000 7f836f7fe700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 0x7f83580a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:10.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.174+0000 7f836f7fe700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 0x7f83580a4000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8364005950 tx=0x7f83640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:10.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.178+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8360097840 con 0x7f8370102010 2024-03-16T20:33:10.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:10 smithi189 ceph-mon[36655]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:10.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:10 smithi023 ceph-mon[31879]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:10.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.395+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f8370075a30 con 0x7f83580a1b40 2024-03-16T20:33:10.408 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.403+0000 7f836d7fa700 1 -- 172.21.15.189:0/1139988793 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8370075a30 con 0x7f83580a1b40 2024-03-16T20:33:10.409 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mds.all update... 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 msgr2=0x7f83580a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 0x7f83580a4000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8364005950 tx=0x7f83640058e0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 msgr2=0x7f837019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8360003930 tx=0x7f836000f690 comp rx=0 tx=0).stop 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 shutdown_connections 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f83580a1b40 0x7f83580a4000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.411 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8370103bb0 0x7f83701a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8370102010 0x7f837019a210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 --2- 172.21.15.189:0/1139988793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8370103210 0x7f837019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:10.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 >> 172.21.15.189:0/1139988793 conn(0x7f83700fd5f0 msgr2=0x7f8370074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:10.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 shutdown_connections 2024-03-16T20:33:10.412 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-16T20:33:10.406+0000 7f837679f700 1 -- 172.21.15.189:0/1139988793 wait complete. 2024-03-16T20:33:10.441 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:10 smithi031 ceph-mon[36281]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:11.067 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.a.service 2024-03-16T20:33:11.071 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.d.service 2024-03-16T20:33:11.114 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.g.service 2024-03-16T20:33:11.157 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.j.service 2024-03-16T20:33:11.200 DEBUG:teuthology.orchestra.run.smithi031:mds.b> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.b.service 2024-03-16T20:33:11.202 DEBUG:teuthology.orchestra.run.smithi031:mds.e> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.e.service 2024-03-16T20:33:11.245 DEBUG:teuthology.orchestra.run.smithi031:mds.h> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.h.service 2024-03-16T20:33:11.288 DEBUG:teuthology.orchestra.run.smithi031:mds.k> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.k.service 2024-03-16T20:33:11.331 DEBUG:teuthology.orchestra.run.smithi189:mds.c> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.c.service 2024-03-16T20:33:11.333 DEBUG:teuthology.orchestra.run.smithi189:mds.f> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.f.service 2024-03-16T20:33:11.376 DEBUG:teuthology.orchestra.run.smithi189:mds.i> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.i.service 2024-03-16T20:33:11.419 DEBUG:teuthology.orchestra.run.smithi189:mds.l> sudo journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.l.service 2024-03-16T20:33:11.462 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-16T20:33:11.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:33:11.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 -- 172.21.15.23:0/1420526322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f281010c0 msgr2=0x7f6f28108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 --2- 172.21.15.23:0/1420526322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f281010c0 0x7f6f28108960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009cd0 tx=0x7f6f2002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 -- 172.21.15.23:0/1420526322 shutdown_connections 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 --2- 172.21.15.23:0/1420526322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f281010c0 0x7f6f28108960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 --2- 172.21.15.23:0/1420526322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28100700 0x7f6f28100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 --2- 172.21.15.23:0/1420526322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f28101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.587+0000 7f6f3000e640 1 -- 172.21.15.23:0/1420526322 >> 172.21.15.23:0/1420526322 conn(0x7f6f280fb130 msgr2=0x7f6f280fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- 172.21.15.23:0/1420526322 shutdown_connections 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- 172.21.15.23:0/1420526322 wait complete. 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 Processor -- start 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- start start 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f281010c0 0x7f6f28102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f281030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28103800 con 0x7f6f28100700 2024-03-16T20:33:11.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28103970 con 0x7f6f281019d0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f3000e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a1620 con 0x7f6f281010c0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f2ddaa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f2ddaa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37136/0 (socket says 172.21.15.23:37136) 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f2ddaa640 1 -- 172.21.15.23:0/3576166506 learned_addr learned my addr 172.21.15.23:0/3576166506 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.588+0000 7f6f2d5a9640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f281010c0 0x7f6f28102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2e5ab640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f281030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 -- 172.21.15.23:0/3576166506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f281010c0 msgr2=0x7f6f28102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f281010c0 0x7f6f28102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 -- 172.21.15.23:0/3576166506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 msgr2=0x7f6f281030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f281030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 -- 172.21.15.23:0/3576166506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f20009900 con 0x7f6f28100700 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2e5ab640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f281030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f2ddaa640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400d7a0 tx=0x7f6f2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24009c30 con 0x7f6f28100700 2024-03-16T20:33:11.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f2400deb0 con 0x7f6f28100700 2024-03-16T20:33:11.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f281a18d0 con 0x7f6f28100700 2024-03-16T20:33:11.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24010460 con 0x7f6f28100700 2024-03-16T20:33:11.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.589+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281a1de0 con 0x7f6f28100700 2024-03-16T20:33:11.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.590+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2810be70 con 0x7f6f28100700 2024-03-16T20:33:11.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.592+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6f24003ef0 con 0x7f6f28100700 2024-03-16T20:33:11.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.592+0000 7f6f1effd640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.592+0000 7f6f2d5a9640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.592+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f240ccd00 con 0x7f6f28100700 2024-03-16T20:33:11.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.593+0000 7f6f2d5a9640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f28104290 tx=0x7f6f18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:11.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.596+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6f240963b0 con 0x7f6f28100700 2024-03-16T20:33:11.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:33:11.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:11.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:11 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:33:11.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-03-16T20:33:11.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.691 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:11.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:11 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.779+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f281a1f70 con 0x7f6f28100700 2024-03-16T20:33:11.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.780+0000 7f6f1effd640 1 -- 172.21.15.23:0/3576166506 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f6f24020090 con 0x7f6f28100700 2024-03-16T20:33:11.785 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:11.785 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:32:59.716148+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 msgr2=0x7f6f140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f28104290 tx=0x7f6f18005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 msgr2=0x7f6f28104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400d7a0 tx=0x7f6f2400dc60 comp rx=0 tx=0).stop 2024-03-16T20:33:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 shutdown_connections 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f281019d0 0x7f6f281030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f281010c0 0x7f6f28102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 --2- 172.21.15.23:0/3576166506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28100700 0x7f6f28104520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 >> 172.21.15.23:0/3576166506 conn(0x7f6f280fb130 msgr2=0x7f6f280ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 shutdown_connections 2024-03-16T20:33:11.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.782+0000 7f6f3000e640 1 -- 172.21.15.23:0/3576166506 wait complete. 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:11.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:11.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:11.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:11.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:11 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:11.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T20:33:11.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:33:11.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- 172.21.15.23:0/1172527694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410103220 msgr2=0x7f4410109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 --2- 172.21.15.23:0/1172527694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410103220 0x7f4410109ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4408009ca0 tx=0x7f440802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- 172.21.15.23:0/1172527694 shutdown_connections 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 --2- 172.21.15.23:0/1172527694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410103220 0x7f4410109ab0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 --2- 172.21.15.23:0/1172527694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410102860 0x7f4410102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 --2- 172.21.15.23:0/1172527694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f44100ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- 172.21.15.23:0/1172527694 >> 172.21.15.23:0/1172527694 conn(0x7f44100fb0e0 msgr2=0x7f44100fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- 172.21.15.23:0/1172527694 shutdown_connections 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- 172.21.15.23:0/1172527694 wait complete. 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 Processor -- start 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 -- start start 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.900+0000 7f441712c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f4410197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 0x7f441019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019f080 con 0x7f4410102860 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019f1f0 con 0x7f44100ff7f0 2024-03-16T20:33:11.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019f3b0 con 0x7f4410103220 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37160/0 (socket says 172.21.15.23:37160) 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 -- 172.21.15.23:0/2640208136 learned_addr learned my addr 172.21.15.23:0/2640208136 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f44156c9640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 0x7f441019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f4414ec8640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f4410197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 -- 172.21.15.23:0/2640208136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 msgr2=0x7f441019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 0x7f441019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 -- 172.21.15.23:0/2640208136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 msgr2=0x7f4410197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f4410197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 -- 172.21.15.23:0/2640208136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408009900 con 0x7f4410102860 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f44156c9640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 0x7f441019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f4414ec8640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f4410197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440ffff640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00e9f0 tx=0x7f43fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc009720 con 0x7f4410102860 2024-03-16T20:33:11.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43fc00cd80 con 0x7f4410102860 2024-03-16T20:33:11.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441019f690 con 0x7f4410102860 2024-03-16T20:33:11.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc010690 con 0x7f4410102860 2024-03-16T20:33:11.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.901+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441019fb50 con 0x7f4410102860 2024-03-16T20:33:11.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.902+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4410108990 con 0x7f4410102860 2024-03-16T20:33:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.903+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f43fc0046c0 con 0x7f4410102860 2024-03-16T20:33:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.904+0000 7f440dffb640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 0x7f44000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.904+0000 7f4414ec8640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 0x7f44000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.904+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43fc014070 con 0x7f4410102860 2024-03-16T20:33:11.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.904+0000 7f4414ec8640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 0x7f44000a3ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4410069800 tx=0x7f4404005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:11.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:11.908+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f43fc096ad0 con 0x7f4410102860 2024-03-16T20:33:12.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.089+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f441002d000 con 0x7f4410102860 2024-03-16T20:33:12.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.089+0000 7f440dffb640 1 -- 172.21.15.23:0/2640208136 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43fc09a190 con 0x7f4410102860 2024-03-16T20:33:12.095 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:33:12.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 msgr2=0x7f44000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 0x7f44000a3ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4410069800 tx=0x7f4404005e70 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 msgr2=0x7f4410197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00e9f0 tx=0x7f43fc00edf0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 shutdown_connections 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44000a1a30 0x7f44000a3ef0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4410103220 0x7f441019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4410102860 0x7f4410197ad0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 --2- 172.21.15.23:0/2640208136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44100ff7f0 0x7f4410197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 >> 172.21.15.23:0/2640208136 conn(0x7f44100fb0e0 msgr2=0x7f4410105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 shutdown_connections 2024-03-16T20:33:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.091+0000 7f441712c640 1 -- 172.21.15.23:0/2640208136 wait complete. 2024-03-16T20:33:12.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 -- 172.21.15.23:0/3209421025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441071f0 msgr2=0x7fc844109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 --2- 172.21.15.23:0/3209421025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441071f0 0x7fc844109640 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009ca0 tx=0x7fc83c02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 -- 172.21.15.23:0/3209421025 shutdown_connections 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 --2- 172.21.15.23:0/3209421025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441071f0 0x7fc844109640 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 --2- 172.21.15.23:0/3209421025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441048c0 0x7fc844106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 --2- 172.21.15.23:0/3209421025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 0x7fc844104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 -- 172.21.15.23:0/3209421025 >> 172.21.15.23:0/3209421025 conn(0x7fc8440fb980 msgr2=0x7fc8440fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:12.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 -- 172.21.15.23:0/3209421025 shutdown_connections 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.204+0000 7fc84add0640 1 -- 172.21.15.23:0/3209421025 wait complete. 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 Processor -- start 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 -- start start 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 0x7fc844197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 0x7fc84419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84419f050 con 0x7fc8441048c0 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84419f1c0 con 0x7fc844101f90 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84419f380 con 0x7fc8441071f0 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37174/0 (socket says 172.21.15.23:37174) 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 -- 172.21.15.23:0/2835878779 learned_addr learned my addr 172.21.15.23:0/2835878779 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc848b6c640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 0x7fc844197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84936d640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 0x7fc84419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:12.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 -- 172.21.15.23:0/2835878779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 msgr2=0x7fc84419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 0x7fc84419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 -- 172.21.15.23:0/2835878779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 msgr2=0x7fc844197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 0x7fc844197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 -- 172.21.15.23:0/2835878779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c009900 con 0x7fc8441048c0 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84936d640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 0x7fc84419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc843fff640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d740 tx=0x7fc83000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:12.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830009c30 con 0x7fc8441048c0 2024-03-16T20:33:12.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc83000de50 con 0x7fc8441048c0 2024-03-16T20:33:12.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830010460 con 0x7fc8441048c0 2024-03-16T20:33:12.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.205+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84419f660 con 0x7fc8441048c0 2024-03-16T20:33:12.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.206+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84419fb20 con 0x7fc8441048c0 2024-03-16T20:33:12.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.207+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8440fd470 con 0x7fc8441048c0 2024-03-16T20:33:12.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.208+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc830004f10 con 0x7fc8441048c0 2024-03-16T20:33:12.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.208+0000 7fc841ffb640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 0x7fc8340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:12.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.208+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8300cca00 con 0x7fc8441048c0 2024-03-16T20:33:12.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.208+0000 7fc848b6c640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 0x7fc8340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:12.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.209+0000 7fc848b6c640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 0x7fc8340a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc8380058c0 tx=0x7fc83800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:12.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.213+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc8300960b0 con 0x7fc8441048c0 2024-03-16T20:33:12.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.434+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8440fcee0 con 0x7fc8441048c0 2024-03-16T20:33:12.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.435+0000 7fc841ffb640 1 -- 172.21.15.23:0/2835878779 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fc830020230 con 0x7fc8441048c0 2024-03-16T20:33:12.440 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:12.440 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-16T20:33:12.440 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2024-03-16T20:33:12.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 msgr2=0x7fc8340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 0x7fc8340a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc8380058c0 tx=0x7fc83800a4c0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 msgr2=0x7fc844197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d740 tx=0x7fc83000dc00 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 shutdown_connections 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8340a1a30 0x7fc8340a3ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8441071f0 0x7fc84419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8441048c0 0x7fc844197aa0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 --2- 172.21.15.23:0/2835878779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc844101f90 0x7fc844197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 >> 172.21.15.23:0/2835878779 conn(0x7fc8440fb980 msgr2=0x7fc844102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 shutdown_connections 2024-03-16T20:33:12.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:12.437+0000 7fc84add0640 1 -- 172.21.15.23:0/2835878779 wait complete. 2024-03-16T20:33:12.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:12 smithi031 ceph-mon[36281]: Deploying daemon mds.l on smithi189 2024-03-16T20:33:12.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:12 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3576166506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:12.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:12 smithi031 ceph-mon[36281]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:12.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:12 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2640208136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:12.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:12 smithi189 ceph-mon[36655]: Deploying daemon mds.l on smithi189 2024-03-16T20:33:12.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:12 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3576166506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:12.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:12 smithi189 ceph-mon[36655]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:12.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:12 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2640208136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:12.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:12 smithi023 ceph-mon[31879]: Deploying daemon mds.l on smithi189 2024-03-16T20:33:12.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:12 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3576166506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:12.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:12 smithi023 ceph-mon[31879]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:12.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:12 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2640208136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:13.222 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 systemd[1]: Starting Ceph mds.l for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:13.594 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:13 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2835878779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:13.594 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 podman[66843]: 2024-03-16 20:33:13.195538923 +0000 UTC m=+0.017714096 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:13.594 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 podman[66843]: 2024-03-16 20:33:13.311569233 +0000 UTC m=+0.133744439 container create 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:33:13.594 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 podman[66843]: 2024-03-16 20:33:13.505743263 +0000 UTC m=+0.327918427 container init 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T20:33:13.594 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 podman[66843]: 2024-03-16 20:33:13.508484639 +0000 UTC m=+0.330659808 container start 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-16T20:33:13.595 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 ceph-mds[66862]: -- 172.21.15.189:0/569636259 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5579948941a0 con 0x55799486a800 2024-03-16T20:33:13.595 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: starting mds.l at 2024-03-16T20:33:13.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:13 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2835878779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:13.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:13 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2835878779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:13.971 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 bash[66843]: 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 2024-03-16T20:33:13.971 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:33:13 smithi189 systemd[1]: Started Ceph mds.l for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:14.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.565+0000 7f3928727640 1 -- 172.21.15.23:0/2489405325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101600 msgr2=0x7f3920101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.565+0000 7f3928727640 1 --2- 172.21.15.23:0/2489405325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101600 0x7f3920101a80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3910009900 tx=0x7f391002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- 172.21.15.23:0/2489405325 shutdown_connections 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- 172.21.15.23:0/2489405325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920101fc0 0x7f3920108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- 172.21.15.23:0/2489405325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101600 0x7f3920101a80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- 172.21.15.23:0/2489405325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920100410 0x7f3920100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- 172.21.15.23:0/2489405325 >> 172.21.15.23:0/2489405325 conn(0x7f39200fb980 msgr2=0x7f39200fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:14.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- 172.21.15.23:0/2489405325 shutdown_connections 2024-03-16T20:33:14.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- 172.21.15.23:0/2489405325 wait complete. 2024-03-16T20:33:14.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 Processor -- start 2024-03-16T20:33:14.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- start start 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920100410 0x7f39201964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920101600 0x7f3920196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019dfc0 con 0x7f3920101fc0 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019e130 con 0x7f3920101600 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3928727640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019e2f0 con 0x7f3920100410 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3926cc4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.566+0000 7f3926cc4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37194/0 (socket says 172.21.15.23:37194) 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 -- 172.21.15.23:0/4009878096 learned_addr learned my addr 172.21.15.23:0/4009878096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:14.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3925cc2640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920101600 0x7f3920196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f39264c3640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920100410 0x7f39201964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 -- 172.21.15.23:0/4009878096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920100410 msgr2=0x7f39201964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920100410 0x7f39201964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 -- 172.21.15.23:0/4009878096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920101600 msgr2=0x7f3920196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920101600 0x7f3920196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 -- 172.21.15.23:0/4009878096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910009560 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3926cc4640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f391c00eb10 tx=0x7f391c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c004b20 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f391c004c80 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392019e5d0 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c010650 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.567+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392019eab0 con 0x7f3920101fc0 2024-03-16T20:33:14.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.568+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3920107730 con 0x7f3920101fc0 2024-03-16T20:33:14.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.569+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f391c005020 con 0x7f3920101fc0 2024-03-16T20:33:14.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.570+0000 7f39177fe640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 0x7f390c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:14.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.570+0000 7f39264c3640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 0x7f390c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:14.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.570+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f391c014070 con 0x7f3920101fc0 2024-03-16T20:33:14.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.570+0000 7f39264c3640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 0x7f390c0a3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3918005fd0 tx=0x7f3918005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:14.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.574+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f391c096c10 con 0x7f3920101fc0 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:boot 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: fsmap 1 up:standby 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: Deploying daemon mds.k on smithi031 2024-03-16T20:33:14.612 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:14 smithi031 ceph-mon[36281]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:14.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.796+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f392010be20 con 0x7f3920101fc0 2024-03-16T20:33:14.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.797+0000 7f39177fe640 1 -- 172.21.15.23:0/4009878096 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f391c09a2d0 con 0x7f3920101fc0 2024-03-16T20:33:14.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:14.803 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T20:33:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2024-03-16T20:33:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 msgr2=0x7f390c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 0x7f390c0a3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3918005fd0 tx=0x7f3918005850 comp rx=0 tx=0).stop 2024-03-16T20:33:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 msgr2=0x7f392019da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f391c00eb10 tx=0x7f391c00c2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 shutdown_connections 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f390c0a1a30 0x7f390c0a3ef0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3920101fc0 0x7f392019da80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3920101600 0x7f3920196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 --2- 172.21.15.23:0/4009878096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3920100410 0x7f39201964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 >> 172.21.15.23:0/4009878096 conn(0x7f39200fb980 msgr2=0x7f3920104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 shutdown_connections 2024-03-16T20:33:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:14.799+0000 7f3928727640 1 -- 172.21.15.23:0/4009878096 wait complete. 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:boot 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: fsmap 1 up:standby 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:14.814 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:14.815 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: Deploying daemon mds.k on smithi031 2024-03-16T20:33:14.815 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:14 smithi023 ceph-mon[31879]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:14.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:boot 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: fsmap 1 up:standby 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: Deploying daemon mds.k on smithi031 2024-03-16T20:33:14.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:14 smithi189 ceph-mon[36655]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:15.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4009878096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:15.939 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:15 smithi031 systemd[1]: Starting Ceph mds.k for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:15.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4009878096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:16.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4009878096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:16.311 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 podman[65861]: 2024-03-16 20:33:15.936366719 +0000 UTC m=+0.018464122 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:16.311 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 podman[65861]: 2024-03-16 20:33:16.054137422 +0000 UTC m=+0.136234808 container create 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T20:33:16.311 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 podman[65861]: 2024-03-16 20:33:16.306617409 +0000 UTC m=+0.388714800 container init 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-16T20:33:16.689 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 podman[65861]: 2024-03-16 20:33:16.309790916 +0000 UTC m=+0.391888312 container start 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T20:33:16.689 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mds[65880]: -- 172.21.15.31:0/132329262 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2879361a0 con 0x55f28790cc00 2024-03-16T20:33:16.689 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: starting mds.k at 2024-03-16T20:33:16.689 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 bash[65861]: 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 2024-03-16T20:33:16.689 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:33:16 smithi031 systemd[1]: Started Ceph mds.k for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:16.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.914+0000 7fee3d91e640 1 -- 172.21.15.23:0/3223932475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee38100760 msgr2=0x7fee38100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.914+0000 7fee3d91e640 1 --2- 172.21.15.23:0/3223932475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee38100760 0x7fee38100bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fee28009900 tx=0x7fee2802bcd0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 -- 172.21.15.23:0/3223932475 shutdown_connections 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 --2- 172.21.15.23:0/3223932475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 0x7fee38101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 --2- 172.21.15.23:0/3223932475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee38100760 0x7fee38100bc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 --2- 172.21.15.23:0/3223932475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee380ffe00 0x7fee38100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 -- 172.21.15.23:0/3223932475 >> 172.21.15.23:0/3223932475 conn(0x7fee380fb980 msgr2=0x7fee380fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:16.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 -- 172.21.15.23:0/3223932475 shutdown_connections 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 -- 172.21.15.23:0/3223932475 wait complete. 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 Processor -- start 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.915+0000 7fee3d91e640 1 -- start start 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee38100760 0x7fee38199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 0x7fee381a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee381a12b0 con 0x7fee380ffe00 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee381a1420 con 0x7fee38100760 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee381a15e0 con 0x7fee38101100 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37202/0 (socket says 172.21.15.23:37202) 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 -- 172.21.15.23:0/2529861068 learned_addr learned my addr 172.21.15.23:0/2529861068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee367fc640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee38100760 0x7fee38199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:16.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee377fe640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 0x7fee381a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 -- 172.21.15.23:0/2529861068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 msgr2=0x7fee381a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 0x7fee381a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 -- 172.21.15.23:0/2529861068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee38100760 msgr2=0x7fee38199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee38100760 0x7fee38199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 -- 172.21.15.23:0/2529861068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009560 con 0x7fee380ffe00 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee36ffd640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00d950 tx=0x7fee2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c009c30 con 0x7fee380ffe00 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee381a18c0 con 0x7fee380ffe00 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee2c009d90 con 0x7fee380ffe00 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00f7e0 con 0x7fee380ffe00 2024-03-16T20:33:16.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.916+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee381a1da0 con 0x7fee380ffe00 2024-03-16T20:33:16.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.918+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee380682e0 con 0x7fee380ffe00 2024-03-16T20:33:16.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.918+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fee2c010ad0 con 0x7fee380ffe00 2024-03-16T20:33:16.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.919+0000 7fee3c91c640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 0x7fee200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:16.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.919+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fee2c0ccc00 con 0x7fee380ffe00 2024-03-16T20:33:16.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.919+0000 7fee367fc640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 0x7fee200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:16.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.920+0000 7fee367fc640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 0x7fee200a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee28000c00 tx=0x7fee2802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:16.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:16.924+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fee2c0d3080 con 0x7fee380ffe00 2024-03-16T20:33:17.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:17.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:16 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:17.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.168+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee3810ace0 con 0x7fee380ffe00 2024-03-16T20:33:17.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.169+0000 7fee3c91c640 1 -- 172.21.15.23:0/2529861068 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fee2c0962b0 con 0x7fee380ffe00 2024-03-16T20:33:17.174 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:17.174 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T20:33:17.175 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-16T20:33:17.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 msgr2=0x7fee200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:17.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 0x7fee200a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee28000c00 tx=0x7fee2802c880 comp rx=0 tx=0).stop 2024-03-16T20:33:17.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 msgr2=0x7fee381997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:17.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00d950 tx=0x7fee2c00de10 comp rx=0 tx=0).stop 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 shutdown_connections 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fee200a1a30 0x7fee200a3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee38101100 0x7fee381a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee38100760 0x7fee38199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 --2- 172.21.15.23:0/2529861068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee380ffe00 0x7fee381997c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 >> 172.21.15.23:0/2529861068 conn(0x7fee380fb980 msgr2=0x7fee380fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 shutdown_connections 2024-03-16T20:33:17.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:17.171+0000 7fee3d91e640 1 -- 172.21.15.23:0/2529861068 wait complete. 2024-03-16T20:33:17.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:17.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:16 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:17.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:17.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:17.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:17.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:17.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:16 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:17.787 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:17 smithi031 ceph-mon[36281]: Deploying daemon mds.j on smithi023 2024-03-16T20:33:17.787 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:17 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:boot 2024-03-16T20:33:17.787 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:17 smithi031 ceph-mon[36281]: fsmap 2 up:standby 2024-03-16T20:33:17.787 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:17 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:33:17.787 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:17 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2529861068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:17.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:17 smithi023 ceph-mon[31879]: Deploying daemon mds.j on smithi023 2024-03-16T20:33:17.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:17 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:boot 2024-03-16T20:33:17.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:17 smithi023 ceph-mon[31879]: fsmap 2 up:standby 2024-03-16T20:33:17.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:17 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:33:17.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:17 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2529861068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:18.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:17 smithi189 ceph-mon[36655]: Deploying daemon mds.j on smithi023 2024-03-16T20:33:18.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:17 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:boot 2024-03-16T20:33:18.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:17 smithi189 ceph-mon[36655]: fsmap 2 up:standby 2024-03-16T20:33:18.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:17 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:33:18.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:17 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2529861068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:18.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:18 smithi023 ceph-mon[31879]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:19.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:19.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:18 smithi031 ceph-mon[36281]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:19.214 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 podman[69429]: 2024-03-16 20:33:19.167688097 +0000 UTC m=+0.139740370 container create a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T20:33:19.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:18 smithi189 ceph-mon[36655]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:19.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 -- 172.21.15.23:0/2390008059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 msgr2=0x7fa7941041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:19.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 --2- 172.21.15.23:0/2390008059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa7941041a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa784009900 tx=0x7fa78402bc50 comp rx=0 tx=0).stop 2024-03-16T20:33:19.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 -- 172.21.15.23:0/2390008059 shutdown_connections 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 --2- 172.21.15.23:0/2390008059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa794107010 0x7fa794109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 --2- 172.21.15.23:0/2390008059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 0x7fa794106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 --2- 172.21.15.23:0/2390008059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa7941041a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 -- 172.21.15.23:0/2390008059 >> 172.21.15.23:0/2390008059 conn(0x7fa7940fb7a0 msgr2=0x7fa7940fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 -- 172.21.15.23:0/2390008059 shutdown_connections 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 -- 172.21.15.23:0/2390008059 wait complete. 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.291+0000 7fa798953640 1 Processor -- start 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 -- start start 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 0x7fa7940ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa794107010 0x7fa7941002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794100a00 con 0x7fa794101db0 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794100b70 con 0x7fa794107010 2024-03-16T20:33:19.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa798953640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7941a1520 con 0x7fa7941046e0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37226/0 (socket says 172.21.15.23:37226) 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 -- 172.21.15.23:0/290642916 learned_addr learned my addr 172.21.15.23:0/290642916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa792ffd640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 0x7fa7940ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 -- 172.21.15.23:0/290642916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 msgr2=0x7fa7940ffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 0x7fa7940ffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 -- 172.21.15.23:0/290642916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa794107010 msgr2=0x7fa7941002a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa794107010 0x7fa7941002a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 -- 172.21.15.23:0/290642916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784009560 con 0x7fa794101db0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.292+0000 7fa7937fe640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa784006010 tx=0x7fa7840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.293+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7840044c0 con 0x7fa794101db0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.293+0000 7fa798953640 1 -- 172.21.15.23:0/290642916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7941a1770 con 0x7fa794101db0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.293+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7840052d0 con 0x7fa794101db0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.293+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784003e10 con 0x7fa794101db0 2024-03-16T20:33:19.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.293+0000 7fa798953640 1 -- 172.21.15.23:0/290642916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7941a1d00 con 0x7fa794101db0 2024-03-16T20:33:19.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.294+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79410cde0 con 0x7fa794101db0 2024-03-16T20:33:19.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.295+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa784005440 con 0x7fa794101db0 2024-03-16T20:33:19.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.295+0000 7fa790ff9640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 0x7fa77c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:19.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.295+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa7840ef3c0 con 0x7fa794101db0 2024-03-16T20:33:19.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.295+0000 7fa792ffd640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 0x7fa77c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:19.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.296+0000 7fa792ffd640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 0x7fa77c0a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa794101490 tx=0x7fa788005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:19.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.303+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa7840b8a70 con 0x7fa794101db0 2024-03-16T20:33:19.500 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 podman[69429]: 2024-03-16 20:33:19.403991568 +0000 UTC m=+0.376043814 container init a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T20:33:19.500 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 podman[69429]: 2024-03-16 20:33:19.406960571 +0000 UTC m=+0.379012803 container start a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T20:33:19.500 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 ceph-mds[69488]: -- 172.21.15.23:0/3928870876 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be1913c1a0 con 0x55be19113000 2024-03-16T20:33:19.500 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j[69466]: starting mds.j at 2024-03-16T20:33:19.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.552+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7940620f0 con 0x7fa794101db0 2024-03-16T20:33:19.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.552+0000 7fa790ff9640 1 -- 172.21.15.23:0/290642916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fa7840bc130 con 0x7fa794101db0 2024-03-16T20:33:19.559 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:19.559 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T20:33:19.559 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 msgr2=0x7fa77c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 0x7fa77c0a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa794101490 tx=0x7fa788005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 msgr2=0x7fa794101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa784006010 tx=0x7fa7840040c0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 shutdown_connections 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa77c0a1a30 0x7fa77c0a3ef0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa794107010 0x7fa7941002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7941046e0 0x7fa7940ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 --2- 172.21.15.23:0/290642916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa794101db0 0x7fa794101720 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.554+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 >> 172.21.15.23:0/290642916 conn(0x7fa7940fb7a0 msgr2=0x7fa794102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:19.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.555+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 shutdown_connections 2024-03-16T20:33:19.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:19.555+0000 7fa77a7fc640 1 -- 172.21.15.23:0/290642916 wait complete. 2024-03-16T20:33:19.840 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 bash[69429]: a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f 2024-03-16T20:33:19.840 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:33:19 smithi023 systemd[1]: Started Ceph mds.j for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/290642916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: Deploying daemon mds.d on smithi023 2024-03-16T20:33:20.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6834/2314600527,v1:172.21.15.23:6835/2314600527] up:boot 2024-03-16T20:33:20.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: fsmap 3 up:standby 2024-03-16T20:33:20.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T20:33:20.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:20 smithi023 ceph-mon[31879]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/290642916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: Deploying daemon mds.d on smithi023 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6834/2314600527,v1:172.21.15.23:6835/2314600527] up:boot 2024-03-16T20:33:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: fsmap 3 up:standby 2024-03-16T20:33:20.940 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T20:33:20.940 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:20 smithi031 ceph-mon[36281]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/290642916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: Deploying daemon mds.d on smithi023 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6834/2314600527,v1:172.21.15.23:6835/2314600527] up:boot 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: fsmap 3 up:standby 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-16T20:33:20.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:20 smithi189 ceph-mon[36655]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:21.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:21.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 -- 172.21.15.23:0/3169471604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c1093d0 msgr2=0x7fd18c10b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 --2- 172.21.15.23:0/3169471604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c1093d0 0x7fd18c10b860 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd184009be0 tx=0x7fd18402bf80 comp rx=0 tx=0).stop 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 -- 172.21.15.23:0/3169471604 shutdown_connections 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 --2- 172.21.15.23:0/3169471604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c1093d0 0x7fd18c10b860 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 --2- 172.21.15.23:0/3169471604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c073ed0 0x7fd18c074330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 --2- 172.21.15.23:0/3169471604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c0754b0 0x7fd18c073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 -- 172.21.15.23:0/3169471604 >> 172.21.15.23:0/3169471604 conn(0x7fd18c0fcad0 msgr2=0x7fd18c0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 -- 172.21.15.23:0/3169471604 shutdown_connections 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.716+0000 7fd191772640 1 -- 172.21.15.23:0/3169471604 wait complete. 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 Processor -- start 2024-03-16T20:33:21.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 -- start start 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c0754b0 0x7fd18c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c1093d0 0x7fd18c1a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c1a1220 con 0x7fd18c073ed0 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c1a1390 con 0x7fd18c0754b0 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd191772640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c1a1550 con 0x7fd18c1093d0 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33488/0 (socket says 172.21.15.23:33488) 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 -- 172.21.15.23:0/833927694 learned_addr learned my addr 172.21.15.23:0/833927694 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18b7fe640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c1093d0 0x7fd18c1a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 -- 172.21.15.23:0/833927694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c1093d0 msgr2=0x7fd18c1a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c1093d0 0x7fd18c1a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 -- 172.21.15.23:0/833927694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c0754b0 msgr2=0x7fd18c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c0754b0 0x7fd18c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.717+0000 7fd18affd640 1 -- 172.21.15.23:0/833927694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd184009840 con 0x7fd18c073ed0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd18affd640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00e9f0 tx=0x7fd17c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c009700 con 0x7fd18c073ed0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c1a1830 con 0x7fd18c073ed0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd17c00cd80 con 0x7fd18c073ed0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c1a1d90 con 0x7fd18c073ed0 2024-03-16T20:33:21.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.718+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c018920 con 0x7fd18c073ed0 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.719+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c108190 con 0x7fd18c073ed0 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.720+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd17c018a80 con 0x7fd18c073ed0 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.721+0000 7fd173fff640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 0x7fd1740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.721+0000 7fd18a7fc640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 0x7fd1740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.721+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd17c014070 con 0x7fd18c073ed0 2024-03-16T20:33:21.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.721+0000 7fd18a7fc640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 0x7fd1740a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1800058c0 tx=0x7fd180005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:21.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.725+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd17c096f70 con 0x7fd18c073ed0 2024-03-16T20:33:21.840 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:21 smithi023 systemd[1]: Starting Ceph mds.d for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:21.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.955+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd18c10f120 con 0x7fd18c073ed0 2024-03-16T20:33:21.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.955+0000 7fd173fff640 1 -- 172.21.15.23:0/833927694 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2910 (secure 0 0 0) 0x7fd17c026b70 con 0x7fd18c073ed0 2024-03-16T20:33:21.962 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:21.962 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T20:33:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 msgr2=0x7fd1740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 0x7fd1740a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1800058c0 tx=0x7fd180005850 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 msgr2=0x7fd18c199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00e9f0 tx=0x7fd17c00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 shutdown_connections 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1740a1a30 0x7fd1740a3ef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd18c1093d0 0x7fd18c1a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd18c0754b0 0x7fd18c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 --2- 172.21.15.23:0/833927694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd18c073ed0 0x7fd18c199730 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 >> 172.21.15.23:0/833927694 conn(0x7fd18c0fcad0 msgr2=0x7fd18c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 shutdown_connections 2024-03-16T20:33:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:21.957+0000 7fd191772640 1 -- 172.21.15.23:0/833927694 wait complete. 2024-03-16T20:33:22.155 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:22 smithi023 ceph-mon[31879]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:22.155 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:22 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/833927694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:22.155 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:21 smithi023 podman[69814]: 2024-03-16 20:33:21.77186036 +0000 UTC m=+0.017882917 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:22.155 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:21 smithi023 podman[69814]: 2024-03-16 20:33:21.90636732 +0000 UTC m=+0.152389878 container create 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T20:33:22.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:22 smithi031 ceph-mon[36281]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:22.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:22 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/833927694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:22.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:22 smithi189 ceph-mon[36655]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:22.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:22 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/833927694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 podman[69814]: 2024-03-16 20:33:22.150533335 +0000 UTC m=+0.396555904 container init 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 podman[69814]: 2024-03-16 20:33:22.153255023 +0000 UTC m=+0.399277580 container start 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 ceph-mds[69835]: -- 172.21.15.23:0/3505309439 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e9a065a1a0 con 0x55e9a0630c00 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d[69831]: starting mds.d at 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 bash[69814]: 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 2024-03-16T20:33:22.591 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:33:22 smithi023 systemd[1]: Started Ceph mds.d for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:23.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: Deploying daemon mds.e on smithi031 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6836/1010201356,v1:172.21.15.23:6837/1010201356] up:boot 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: fsmap 4 up:standby 2024-03-16T20:33:23.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: Deploying daemon mds.e on smithi031 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6836/1010201356,v1:172.21.15.23:6837/1010201356] up:boot 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: fsmap 4 up:standby 2024-03-16T20:33:23.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T20:33:23.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: Deploying daemon mds.e on smithi031 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6836/1010201356,v1:172.21.15.23:6837/1010201356] up:boot 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: fsmap 4 up:standby 2024-03-16T20:33:23.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-16T20:33:23.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- 172.21.15.23:0/956872514 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc100680 msgr2=0x7fd7dc100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 --2- 172.21.15.23:0/956872514 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc100680 0x7fd7dc100b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009900 tx=0x7fd7cc02bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- 172.21.15.23:0/956872514 shutdown_connections 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 --2- 172.21.15.23:0/956872514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc101040 0x7fd7dc1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 --2- 172.21.15.23:0/956872514 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc100680 0x7fd7dc100b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 --2- 172.21.15.23:0/956872514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101a30 0x7fd7dc101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- 172.21.15.23:0/956872514 >> 172.21.15.23:0/956872514 conn(0x7fd7dc0fb110 msgr2=0x7fd7dc0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- 172.21.15.23:0/956872514 shutdown_connections 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- 172.21.15.23:0/956872514 wait complete. 2024-03-16T20:33:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 Processor -- start 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.086+0000 7fd7e34da640 1 -- start start 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc100680 0x7fd7dc197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101040 0x7fd7dc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19f080 con 0x7fd7dc101a30 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19f1f0 con 0x7fd7dc101040 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19f3b0 con 0x7fd7dc100680 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33524/0 (socket says 172.21.15.23:33524) 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 -- 172.21.15.23:0/3713709621 learned_addr learned my addr 172.21.15.23:0/3713709621 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e0a75640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101040 0x7fd7dc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1276640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc100680 0x7fd7dc197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 -- 172.21.15.23:0/3713709621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc100680 msgr2=0x7fd7dc197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc100680 0x7fd7dc197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 -- 172.21.15.23:0/3713709621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101040 msgr2=0x7fd7dc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101040 0x7fd7dc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 -- 172.21.15.23:0/3713709621 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc009560 con 0x7fd7dc101a30 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e1a77640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400eaa0 tx=0x7fd7d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d400cdb0 con 0x7fd7dc101a30 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7d4005020 con 0x7fd7dc101a30 2024-03-16T20:33:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc19f690 con 0x7fd7dc101a30 2024-03-16T20:33:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40116b0 con 0x7fd7dc101a30 2024-03-16T20:33:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.087+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc19fb70 con 0x7fd7dc101a30 2024-03-16T20:33:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.088+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc10be70 con 0x7fd7dc101a30 2024-03-16T20:33:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.090+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd7d4005520 con 0x7fd7dc101a30 2024-03-16T20:33:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.091+0000 7fd7d27fc640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 0x7fd7c80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.091+0000 7fd7e1276640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 0x7fd7c80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.091+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7d40cd550 con 0x7fd7dc101a30 2024-03-16T20:33:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.091+0000 7fd7e1276640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 0x7fd7c80a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.095+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd7d4096c00 con 0x7fd7dc101a30 2024-03-16T20:33:24.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.315+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7dc10c000 con 0x7fd7dc101a30 2024-03-16T20:33:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.320+0000 7fd7d27fc640 1 -- 172.21.15.23:0/3713709621 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7fd7d401fe90 con 0x7fd7dc101a30 2024-03-16T20:33:24.327 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:24.327 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-16T20:33:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2024-03-16T20:33:24.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 msgr2=0x7fd7c80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 0x7fd7c80a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d8005850 comp rx=0 tx=0).stop 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 msgr2=0x7fd7dc19eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400eaa0 tx=0x7fd7d400ef60 comp rx=0 tx=0).stop 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 shutdown_connections 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7c80a1a30 0x7fd7c80a3ef0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7dc101a30 0x7fd7dc19eb40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7dc101040 0x7fd7dc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 --2- 172.21.15.23:0/3713709621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7dc100680 0x7fd7dc197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 >> 172.21.15.23:0/3713709621 conn(0x7fd7dc0fb110 msgr2=0x7fd7dc0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 shutdown_connections 2024-03-16T20:33:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:24.322+0000 7fd7e34da640 1 -- 172.21.15.23:0/3713709621 wait complete. 2024-03-16T20:33:24.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:24 smithi023 ceph-mon[31879]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:24.598 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:24 smithi031 ceph-mon[36281]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:24.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:24 smithi189 ceph-mon[36655]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:24.939 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:24 smithi031 podman[66186]: 2024-03-16 20:33:24.817166795 +0000 UTC m=+0.118294622 container create 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True) 2024-03-16T20:33:25.333 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 podman[66186]: 2024-03-16 20:33:25.053246997 +0000 UTC m=+0.354374836 container init 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T20:33:25.333 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 podman[66186]: 2024-03-16 20:33:25.056046126 +0000 UTC m=+0.357173954 container start 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-16T20:33:25.333 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mds[66205]: -- 172.21.15.31:0/3303193341 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5569f1a7c1a0 con 0x5569f1a53000 2024-03-16T20:33:25.333 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: starting mds.e at 2024-03-16T20:33:25.334 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 bash[66186]: 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 2024-03-16T20:33:25.334 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:33:25 smithi031 systemd[1]: Started Ceph mds.e for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3713709621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:25.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3713709621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:25.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3713709621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:25.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:25.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:25.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:26.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 -- 172.21.15.23:0/1847371603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c107270 msgr2=0x7fda7c1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 --2- 172.21.15.23:0/1847371603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c107270 0x7fda7c1096c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fda74009ca0 tx=0x7fda7402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 -- 172.21.15.23:0/1847371603 shutdown_connections 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 --2- 172.21.15.23:0/1847371603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c107270 0x7fda7c1096c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 --2- 172.21.15.23:0/1847371603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c104940 0x7fda7c106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 --2- 172.21.15.23:0/1847371603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c102010 0x7fda7c104400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 -- 172.21.15.23:0/1847371603 >> 172.21.15.23:0/1847371603 conn(0x7fda7c0fba20 msgr2=0x7fda7c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 -- 172.21.15.23:0/1847371603 shutdown_connections 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.448+0000 7fda81e88640 1 -- 172.21.15.23:0/1847371603 wait complete. 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 Processor -- start 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 -- start start 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c104940 0x7fda7c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c107270 0x7fda7c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c19f050 con 0x7fda7c102010 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c19f1c0 con 0x7fda7c104940 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda81e88640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c19f380 con 0x7fda7c107270 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.449+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33544/0 (socket says 172.21.15.23:33544) 2024-03-16T20:33:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 -- 172.21.15.23:0/3742455815 learned_addr learned my addr 172.21.15.23:0/3742455815 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7affd640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c104940 0x7fda7c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7bfff640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c107270 0x7fda7c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 -- 172.21.15.23:0/3742455815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c107270 msgr2=0x7fda7c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c107270 0x7fda7c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 -- 172.21.15.23:0/3742455815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c104940 msgr2=0x7fda7c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c104940 0x7fda7c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 -- 172.21.15.23:0/3742455815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74009900 con 0x7fda7c102010 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda7b7fe640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00d950 tx=0x7fda6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c009c30 con 0x7fda7c102010 2024-03-16T20:33:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda6c009d90 con 0x7fda7c102010 2024-03-16T20:33:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c00f7e0 con 0x7fda7c102010 2024-03-16T20:33:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c19f660 con 0x7fda7c102010 2024-03-16T20:33:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.450+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c19fb70 con 0x7fda7c102010 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.451+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7c0fd510 con 0x7fda7c102010 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.452+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fda6c010ad0 con 0x7fda7c102010 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.453+0000 7fda78ff9640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 0x7fda640a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.453+0000 7fda7affd640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 0x7fda640a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.453+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda6c0ccb40 con 0x7fda7c102010 2024-03-16T20:33:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.454+0000 7fda7affd640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 0x7fda640a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda7c198a70 tx=0x7fda70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.458+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fda6c0962a0 con 0x7fda7c102010 2024-03-16T20:33:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:26 smithi189 ceph-mon[36655]: Deploying daemon mds.f on smithi189 2024-03-16T20:33:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:26 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6834/354681253,v1:172.21.15.31:6835/354681253] up:boot 2024-03-16T20:33:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:26 smithi189 ceph-mon[36655]: fsmap 5 up:standby 2024-03-16T20:33:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:26 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:33:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:26 smithi189 ceph-mon[36655]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:26.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.681+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda7c10cf30 con 0x7fda7c102010 2024-03-16T20:33:26.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.682+0000 7fda78ff9640 1 -- 172.21.15.23:0/3742455815 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7fda7c10cf30 con 0x7fda7c102010 2024-03-16T20:33:26.688 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:26.688 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-16T20:33:26.689 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2024-03-16T20:33:26.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:26 smithi031 ceph-mon[36281]: Deploying daemon mds.f on smithi189 2024-03-16T20:33:26.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:26 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6834/354681253,v1:172.21.15.31:6835/354681253] up:boot 2024-03-16T20:33:26.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:26 smithi031 ceph-mon[36281]: fsmap 5 up:standby 2024-03-16T20:33:26.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:26 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:33:26.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:26 smithi031 ceph-mon[36281]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:26.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 msgr2=0x7fda640a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 0x7fda640a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda7c198a70 tx=0x7fda70005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 msgr2=0x7fda7c197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00d950 tx=0x7fda6c00de10 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 shutdown_connections 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fda640a1a30 0x7fda640a3ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda7c107270 0x7fda7c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda7c104940 0x7fda7c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 --2- 172.21.15.23:0/3742455815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda7c102010 0x7fda7c197560 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 >> 172.21.15.23:0/3742455815 conn(0x7fda7c0fba20 msgr2=0x7fda7c102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 shutdown_connections 2024-03-16T20:33:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:26.684+0000 7fda81e88640 1 -- 172.21.15.23:0/3742455815 wait complete. 2024-03-16T20:33:26.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:26 smithi023 ceph-mon[31879]: Deploying daemon mds.f on smithi189 2024-03-16T20:33:26.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:26 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6834/354681253,v1:172.21.15.31:6835/354681253] up:boot 2024-03-16T20:33:26.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:26 smithi023 ceph-mon[31879]: fsmap 5 up:standby 2024-03-16T20:33:26.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:33:26.700 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:26 smithi023 ceph-mon[31879]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:27.347 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 systemd[1]: Starting Ceph mds.f for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:27.656 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:27 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3742455815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:27.656 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 podman[67156]: 2024-03-16 20:33:27.345242708 +0000 UTC m=+0.018206048 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:27.656 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 podman[67156]: 2024-03-16 20:33:27.457576954 +0000 UTC m=+0.130540276 container create 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:33:27.656 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 podman[67156]: 2024-03-16 20:33:27.651568296 +0000 UTC m=+0.324531627 container init 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T20:33:27.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:27 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3742455815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:27.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:27 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3742455815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:27.971 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 podman[67156]: 2024-03-16 20:33:27.654317403 +0000 UTC m=+0.327280736 container start 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T20:33:27.971 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 ceph-mds[67175]: -- 172.21.15.189:0/1265217828 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5636387341a0 con 0x56363870b000 2024-03-16T20:33:27.972 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: starting mds.f at 2024-03-16T20:33:27.972 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 bash[67156]: 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea 2024-03-16T20:33:27.972 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:33:27 smithi189 systemd[1]: Started Ceph mds.f for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:28.704 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 -- 172.21.15.23:0/2165601099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 msgr2=0x7f90a01043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/2165601099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a01043a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9090009900 tx=0x7f909002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 -- 172.21.15.23:0/2165601099 shutdown_connections 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/2165601099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 0x7f90a0109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/2165601099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 0x7f90a0106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/2165601099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a01043a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 -- 172.21.15.23:0/2165601099 >> 172.21.15.23:0/2165601099 conn(0x7f90a00fb9a0 msgr2=0x7f90a00fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 -- 172.21.15.23:0/2165601099 shutdown_connections 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.814+0000 7f90a5ff4640 1 -- 172.21.15.23:0/2165601099 wait complete. 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 Processor -- start 2024-03-16T20:33:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 -- start start 2024-03-16T20:33:28.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:28.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 0x7f90a0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:28.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 0x7f90a019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:28.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019f0f0 con 0x7f90a0101fb0 2024-03-16T20:33:28.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019f260 con 0x7f90a01048e0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.815+0000 7f90a5ff4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019f420 con 0x7f90a0107210 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33556/0 (socket says 172.21.15.23:33556) 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 -- 172.21.15.23:0/1239825083 learned_addr learned my addr 172.21.15.23:0/1239825083 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909effd640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 0x7f90a0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909ffff640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 0x7f90a019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 -- 172.21.15.23:0/1239825083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 msgr2=0x7f90a019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 0x7f90a019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 -- 172.21.15.23:0/1239825083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 msgr2=0x7f90a0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 0x7f90a0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 -- 172.21.15.23:0/1239825083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9090009560 con 0x7f90a0101fb0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909f7fe640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9090006010 tx=0x7f90900040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90900044c0 con 0x7f90a0101fb0 2024-03-16T20:33:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a019f6a0 con 0x7f90a0101fb0 2024-03-16T20:33:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90900052d0 con 0x7f90a0101fb0 2024-03-16T20:33:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090003e10 con 0x7f90a0101fb0 2024-03-16T20:33:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.816+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a019fbb0 con 0x7f90a0101fb0 2024-03-16T20:33:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.817+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a00fd490 con 0x7f90a0101fb0 2024-03-16T20:33:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.819+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9090005440 con 0x7f90a0101fb0 2024-03-16T20:33:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.819+0000 7f909cff9640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 0x7f90880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.819+0000 7f909effd640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 0x7f90880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.819+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90900ef830 con 0x7f90a0101fb0 2024-03-16T20:33:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.820+0000 7f909effd640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 0x7f90880a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90a0198b10 tx=0x7f9094005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:28.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:28.823+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f90900b8ee0 con 0x7f90a0101fb0 2024-03-16T20:33:28.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: Deploying daemon mds.c on smithi189 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6834/1658699703,v1:172.21.15.189:6835/1658699703] up:boot 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: fsmap 6 up:standby 2024-03-16T20:33:28.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:28 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:33:29.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.044+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a010cf40 con 0x7f90a0101fb0 2024-03-16T20:33:29.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.045+0000 7f909cff9640 1 -- 172.21.15.23:0/1239825083 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5283 (secure 0 0 0) 0x7f90900bc5a0 con 0x7f90a0101fb0 2024-03-16T20:33:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:29.051 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-16T20:33:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 msgr2=0x7f90880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 0x7f90880a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90a0198b10 tx=0x7f9094005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 msgr2=0x7f90a0197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9090006010 tx=0x7f90900040c0 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 shutdown_connections 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f90880a1a30 0x7f90880a3ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90a0107210 0x7f90a019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90a01048e0 0x7f90a0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 --2- 172.21.15.23:0/1239825083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90a0101fb0 0x7f90a0197600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 >> 172.21.15.23:0/1239825083 conn(0x7f90a00fb9a0 msgr2=0x7f90a0102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 shutdown_connections 2024-03-16T20:33:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:29.047+0000 7f90a5ff4640 1 -- 172.21.15.23:0/1239825083 wait complete. 2024-03-16T20:33:29.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: Deploying daemon mds.c on smithi189 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6834/1658699703,v1:172.21.15.189:6835/1658699703] up:boot 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: fsmap 6 up:standby 2024-03-16T20:33:29.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:28 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:33:29.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:29.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:29.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: Deploying daemon mds.c on smithi189 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6834/1658699703,v1:172.21.15.189:6835/1658699703] up:boot 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: fsmap 6 up:standby 2024-03-16T20:33:29.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:28 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:33:30.067 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:29 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1239825083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:30.068 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 podman[67464]: 2024-03-16 20:33:29.953283953 +0000 UTC m=+0.017777723 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:30.068 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 podman[67464]: 2024-03-16 20:33:30.062621411 +0000 UTC m=+0.127115189 container create 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T20:33:30.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:29 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1239825083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:30.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:29 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1239825083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:30.329 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 podman[67464]: 2024-03-16 20:33:30.298796542 +0000 UTC m=+0.363290301 container init 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-16T20:33:30.329 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 podman[67464]: 2024-03-16 20:33:30.301397385 +0000 UTC m=+0.365891143 container start 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T20:33:30.721 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mds[67483]: -- 172.21.15.189:0/1036216758 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f43c7861a0 con 0x55f43c75cc00 2024-03-16T20:33:30.722 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: starting mds.c at 2024-03-16T20:33:30.722 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 bash[67464]: 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de 2024-03-16T20:33:30.722 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:33:30 smithi189 systemd[1]: Started Ceph mds.c for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:31.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:31.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:31.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:30 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:31.127 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:31.128 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:30 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 -- 172.21.15.23:0/4111096431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108104940 msgr2=0x7f5108106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 --2- 172.21.15.23:0/4111096431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108104940 0x7f5108106d30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009900 tx=0x7f50f802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 -- 172.21.15.23:0/4111096431 shutdown_connections 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 --2- 172.21.15.23:0/4111096431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108107270 0x7f51081096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 --2- 172.21.15.23:0/4111096431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108104940 0x7f5108106d30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 --2- 172.21.15.23:0/4111096431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 0x7f5108104400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.170+0000 7f510e7a1640 1 -- 172.21.15.23:0/4111096431 >> 172.21.15.23:0/4111096431 conn(0x7f51080fba20 msgr2=0x7f51080fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- 172.21.15.23:0/4111096431 shutdown_connections 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- 172.21.15.23:0/4111096431 wait complete. 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 Processor -- start 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- start start 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 0x7f5108197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108104940 0x7f5108197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510819f000 con 0x7f5108107270 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510819f170 con 0x7f5108102010 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510e7a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510819f330 con 0x7f5108104940 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.171+0000 7f510cd3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35096/0 (socket says 172.21.15.23:35096) 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 -- 172.21.15.23:0/2214847927 learned_addr learned my addr 172.21.15.23:0/2214847927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:31.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f5107fff640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 0x7f5108197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f51077fe640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108104940 0x7f5108197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 -- 172.21.15.23:0/2214847927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108104940 msgr2=0x7f5108197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108104940 0x7f5108197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 -- 172.21.15.23:0/2214847927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 msgr2=0x7f5108197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 0x7f5108197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 -- 172.21.15.23:0/2214847927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f8009560 con 0x7f5108107270 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510cd3e640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f510000eaa0 tx=0x7f510000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5100004d20 con 0x7f5108107270 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510819f610 con 0x7f5108107270 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5100004e80 con 0x7f5108107270 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51000116d0 con 0x7f5108107270 2024-03-16T20:33:31.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.172+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510819fb40 con 0x7f5108107270 2024-03-16T20:33:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.173+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51080fd510 con 0x7f5108107270 2024-03-16T20:33:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.174+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f510000caf0 con 0x7f5108107270 2024-03-16T20:33:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.175+0000 7f51057fa640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 0x7f50f00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.175+0000 7f5107fff640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 0x7f50f00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.175+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f51000cd160 con 0x7f5108107270 2024-03-16T20:33:31.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.175+0000 7f5107fff640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 0x7f50f00a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00abe0 tx=0x7f50fc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:31.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.179+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5100096810 con 0x7f5108107270 2024-03-16T20:33:31.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:31.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:31.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:31.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:31.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:30 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.399+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f510810cf30 con 0x7f5108107270 2024-03-16T20:33:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.399+0000 7f51057fa640 1 -- 172.21.15.23:0/2214847927 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7f510810cf30 con 0x7f5108107270 2024-03-16T20:33:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:31.406 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-16T20:33:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-16T20:33:31.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 msgr2=0x7f50f00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:31.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 0x7f50f00a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00abe0 tx=0x7f50fc00a2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 msgr2=0x7f510819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:31.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f510000eaa0 tx=0x7f510000eea0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 shutdown_connections 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f50f00a1a30 0x7f50f00a3ef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5108107270 0x7f510819eac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5108104940 0x7f5108197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 --2- 172.21.15.23:0/2214847927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5108102010 0x7f5108197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 >> 172.21.15.23:0/2214847927 conn(0x7f51080fba20 msgr2=0x7f5108102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 shutdown_connections 2024-03-16T20:33:31.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:31.402+0000 7f510e7a1640 1 -- 172.21.15.23:0/2214847927 wait complete. 2024-03-16T20:33:32.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:31 smithi023 ceph-mon[31879]: Deploying daemon mds.h on smithi031 2024-03-16T20:33:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:31 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6836/3204120502,v1:172.21.15.189:6837/3204120502] up:boot 2024-03-16T20:33:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:31 smithi023 ceph-mon[31879]: fsmap 7 up:standby 2024-03-16T20:33:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:31 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:33:32.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:31 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2214847927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:32.123 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:31 smithi189 ceph-mon[36655]: Deploying daemon mds.h on smithi031 2024-03-16T20:33:32.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:31 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6836/3204120502,v1:172.21.15.189:6837/3204120502] up:boot 2024-03-16T20:33:32.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:31 smithi189 ceph-mon[36655]: fsmap 7 up:standby 2024-03-16T20:33:32.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:31 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:33:32.124 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:31 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2214847927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:32.183 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:31 smithi031 ceph-mon[36281]: Deploying daemon mds.h on smithi031 2024-03-16T20:33:32.183 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:31 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6836/3204120502,v1:172.21.15.189:6837/3204120502] up:boot 2024-03-16T20:33:32.183 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:31 smithi031 ceph-mon[36281]: fsmap 7 up:standby 2024-03-16T20:33:32.183 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:31 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:33:32.183 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:31 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2214847927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:32.832 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:32 smithi031 systemd[1]: Starting Ceph mds.h for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:33.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:32 smithi023 ceph-mon[31879]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:33.154 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:32 smithi031 ceph-mon[36281]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:33.155 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:32 smithi031 podman[66496]: 2024-03-16 20:33:32.89668705 +0000 UTC m=+0.103797542 container create 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-16T20:33:33.155 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:32 smithi031 podman[66496]: 2024-03-16 20:33:32.810810796 +0000 UTC m=+0.017921303 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:33.155 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 podman[66496]: 2024-03-16 20:33:33.149866749 +0000 UTC m=+0.356977255 container init 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T20:33:33.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:32 smithi189 ceph-mon[36655]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:33.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:33.438 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 podman[66496]: 2024-03-16 20:33:33.1535842 +0000 UTC m=+0.360694696 container start 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T20:33:33.439 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 ceph-mds[66516]: -- 172.21.15.31:0/3292453639 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559de9e161a0 con 0x559de9dec800 2024-03-16T20:33:33.439 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: starting mds.h at 2024-03-16T20:33:33.439 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 bash[66496]: 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b 2024-03-16T20:33:33.439 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:33:33 smithi031 systemd[1]: Started Ceph mds.h for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:33.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 -- 172.21.15.23:0/3774731095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101600 msgr2=0x7f4cec101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3774731095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101600 0x7f4cec101a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009900 tx=0x7f4ce002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 -- 172.21.15.23:0/3774731095 shutdown_connections 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3774731095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101fc0 0x7f4cec108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3774731095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101600 0x7f4cec101a80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3774731095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 -- 172.21.15.23:0/3774731095 >> 172.21.15.23:0/3774731095 conn(0x7f4cec0fb980 msgr2=0x7f4cec0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 -- 172.21.15.23:0/3774731095 shutdown_connections 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 -- 172.21.15.23:0/3774731095 wait complete. 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.526+0000 7f4cf153a640 1 Processor -- start 2024-03-16T20:33:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 -- start start 2024-03-16T20:33:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101600 0x7f4cec196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19df30 con 0x7f4cec101fc0 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19e0a0 con 0x7f4cec100410 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cf153a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19e260 con 0x7f4cec101600 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:55530/0 (socket says 172.21.15.23:55530) 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35116/0 (socket says 172.21.15.23:35116) 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 -- 172.21.15.23:0/3658138854 learned_addr learned my addr 172.21.15.23:0/3658138854 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4cea7fc640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101600 0x7f4cec196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 -- 172.21.15.23:0/3658138854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101600 msgr2=0x7f4cec196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101600 0x7f4cec196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 -- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 msgr2=0x7f4cec19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 -- 172.21.15.23:0/3658138854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0009560 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceb7fe640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.527+0000 7f4ceaffd640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9c0 tx=0x7f4cdc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.528+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc009700 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.528+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec19e540 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.528+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc00cd80 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.528+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc010430 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.528+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec19ea20 con 0x7f4cec100410 2024-03-16T20:33:33.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.529+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec107730 con 0x7f4cec100410 2024-03-16T20:33:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.530+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4cdc003680 con 0x7f4cec100410 2024-03-16T20:33:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.530+0000 7f4cd3fff640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 0x7f4cd40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.530+0000 7f4cea7fc640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 0x7f4cd40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.530+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4cdc014070 con 0x7f4cec100410 2024-03-16T20:33:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.531+0000 7f4cea7fc640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 0x7f4cd40a3ef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009230 tx=0x7f4ce0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.534+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4cdc0969b0 con 0x7f4cec100410 2024-03-16T20:33:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.755+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cec10be20 con 0x7f4cec100410 2024-03-16T20:33:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.756+0000 7f4cd3fff640 1 -- 172.21.15.23:0/3658138854 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7f4cdc09a070 con 0x7f4cec100410 2024-03-16T20:33:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:33.761 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-16T20:33:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-16T20:33:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 msgr2=0x7f4cd40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 0x7f4cd40a3ef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009230 tx=0x7f4ce0005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 msgr2=0x7f4cec196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9c0 tx=0x7f4cdc00ee80 comp rx=0 tx=0).stop 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 shutdown_connections 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4cd40a1a30 0x7f4cd40a3ef0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cec101fc0 0x7f4cec19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cec101600 0x7f4cec196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 --2- 172.21.15.23:0/3658138854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cec100410 0x7f4cec196440 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 >> 172.21.15.23:0/3658138854 conn(0x7f4cec0fb980 msgr2=0x7f4cec104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 shutdown_connections 2024-03-16T20:33:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:33.758+0000 7f4cf153a640 1 -- 172.21.15.23:0/3658138854 wait complete. 2024-03-16T20:33:34.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: Deploying daemon mds.i on smithi189 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3658138854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:boot 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: fsmap 8 up:standby 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:33:34.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:34 smithi023 ceph-mon[31879]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:34.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: Deploying daemon mds.i on smithi189 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3658138854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:boot 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: fsmap 8 up:standby 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:33:34.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:34 smithi031 ceph-mon[36281]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:34.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: Deploying daemon mds.i on smithi189 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3658138854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:boot 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: fsmap 8 up:standby 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:33:34.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:34 smithi189 ceph-mon[36655]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:35.681 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 systemd[1]: Starting Ceph mds.i for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:35.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.883+0000 7fb98fdb6640 1 -- 172.21.15.23:0/4024493818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988104920 msgr2=0x7fb988106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.883+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/4024493818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988104920 0x7fb988106d10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb984009900 tx=0x7fb98402bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- 172.21.15.23:0/4024493818 shutdown_connections 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/4024493818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988107250 0x7fb988109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/4024493818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988104920 0x7fb988106d10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/4024493818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988101ff0 0x7fb9881043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- 172.21.15.23:0/4024493818 >> 172.21.15.23:0/4024493818 conn(0x7fb9880fb980 msgr2=0x7fb9880fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- 172.21.15.23:0/4024493818 shutdown_connections 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- 172.21.15.23:0/4024493818 wait complete. 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 Processor -- start 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- start start 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101ff0 0x7fb988197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988104920 0x7fb988197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819f150 con 0x7fb988107250 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819f2c0 con 0x7fb988101ff0 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98fdb6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819f480 con 0x7fb988104920 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.884+0000 7fb98e353640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98d351640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988104920 0x7fb988197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35140/0 (socket says 172.21.15.23:35140) 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 -- 172.21.15.23:0/2049870958 learned_addr learned my addr 172.21.15.23:0/2049870958 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98db52640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101ff0 0x7fb988197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 -- 172.21.15.23:0/2049870958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988104920 msgr2=0x7fb988197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988104920 0x7fb988197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 -- 172.21.15.23:0/2049870958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101ff0 msgr2=0x7fb988197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101ff0 0x7fb988197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 -- 172.21.15.23:0/2049870958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb984009560 con 0x7fb988107250 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98e353640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb98000b970 tx=0x7fb98000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98000cb40 con 0x7fb988107250 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98819f760 con 0x7fb988107250 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb98000cca0 con 0x7fb988107250 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9800077d0 con 0x7fb988107250 2024-03-16T20:33:35.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.885+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98819fbf0 con 0x7fb988107250 2024-03-16T20:33:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.886+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9880fd470 con 0x7fb988107250 2024-03-16T20:33:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.887+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb980014440 con 0x7fb988107250 2024-03-16T20:33:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.888+0000 7fb97affd640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 0x7fb9740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.888+0000 7fb98db52640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 0x7fb9740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:35.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.888+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9800cc820 con 0x7fb988107250 2024-03-16T20:33:35.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.888+0000 7fb98db52640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 0x7fb9740a3ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb97c005fd0 tx=0x7fb97c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:35.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:35.893+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb980095f80 con 0x7fb988107250 2024-03-16T20:33:35.941 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 podman[67787]: 2024-03-16 20:33:35.579416241 +0000 UTC m=+0.017879550 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:35.941 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 podman[67787]: 2024-03-16 20:33:35.684075225 +0000 UTC m=+0.122538534 container create 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T20:33:35.941 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 podman[67787]: 2024-03-16 20:33:35.936830781 +0000 UTC m=+0.375294090 container init 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T20:33:36.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:36 smithi023 ceph-mon[31879]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.114+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb98810cf60 con 0x7fb988107250 2024-03-16T20:33:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.115+0000 7fb97affd640 1 -- 172.21.15.23:0/2049870958 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7661 (secure 0 0 0) 0x7fb980099640 con 0x7fb988107250 2024-03-16T20:33:36.120 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:36.121 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-16T20:33:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-16T20:33:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 msgr2=0x7fb9740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 0x7fb9740a3ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb97c005fd0 tx=0x7fb97c005c70 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 msgr2=0x7fb98819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb98000b970 tx=0x7fb98000be30 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 shutdown_connections 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb9740a1a30 0x7fb9740a3ef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988107250 0x7fb98819ec10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb988104920 0x7fb988197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 --2- 172.21.15.23:0/2049870958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101ff0 0x7fb988197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 >> 172.21.15.23:0/2049870958 conn(0x7fb9880fb980 msgr2=0x7fb988102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 shutdown_connections 2024-03-16T20:33:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:36.117+0000 7fb98fdb6640 1 -- 172.21.15.23:0/2049870958 wait complete. 2024-03-16T20:33:36.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:36 smithi189 ceph-mon[36655]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:36.222 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 podman[67787]: 2024-03-16 20:33:35.939550187 +0000 UTC m=+0.378013499 container start 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-16T20:33:36.222 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 ceph-mds[67806]: -- 172.21.15.189:0/1435893583 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e1084a1a0 con 0x559e10820800 2024-03-16T20:33:36.222 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:35 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: starting mds.i at 2024-03-16T20:33:36.222 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:36 smithi189 bash[67787]: 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 2024-03-16T20:33:36.222 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:36 smithi189 systemd[1]: Started Ceph mds.i for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:36.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:36 smithi031 ceph-mon[36281]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:37.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:boot 2024-03-16T20:33:37.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: fsmap 9 up:standby 2024-03-16T20:33:37.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:33:37.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2049870958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:37.342 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:37 smithi023 ceph-mon[31879]: Deploying daemon mds.g on smithi023 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:boot 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: fsmap 9 up:standby 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2049870958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:37.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:37.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:37.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:37 smithi031 ceph-mon[36281]: Deploying daemon mds.g on smithi023 2024-03-16T20:33:37.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:boot 2024-03-16T20:33:37.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: fsmap 9 up:standby 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2049870958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:37.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:37 smithi189 ceph-mon[36655]: Deploying daemon mds.g on smithi023 2024-03-16T20:33:38.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:38 smithi023 ceph-mon[31879]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:38.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 -- 172.21.15.23:0/2618705351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 msgr2=0x7f33ec104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 --2- 172.21.15.23:0/2618705351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec104440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009900 tx=0x7f33dc02bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 -- 172.21.15.23:0/2618705351 shutdown_connections 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 --2- 172.21.15.23:0/2618705351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 0x7f33ec1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 --2- 172.21.15.23:0/2618705351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 --2- 172.21.15.23:0/2618705351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec104440 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 -- 172.21.15.23:0/2618705351 >> 172.21.15.23:0/2618705351 conn(0x7f33ec0fb980 msgr2=0x7f33ec0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 -- 172.21.15.23:0/2618705351 shutdown_connections 2024-03-16T20:33:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 -- 172.21.15.23:0/2618705351 wait complete. 2024-03-16T20:33:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.368+0000 7f33f258c640 1 Processor -- start 2024-03-16T20:33:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 -- start start 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 0x7f33ec19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec19f0c0 con 0x7f33ec102050 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec19f230 con 0x7f33ec104980 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f258c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec19f3f0 con 0x7f33ec1072b0 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35156/0 (socket says 172.21.15.23:35156) 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 -- 172.21.15.23:0/3073843671 learned_addr learned my addr 172.21.15.23:0/3073843671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33eb7fe640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33f0b29640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 0x7f33ec19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 -- 172.21.15.23:0/3073843671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 msgr2=0x7f33ec19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 0x7f33ec19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 -- 172.21.15.23:0/3073843671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 msgr2=0x7f33ec197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33ebfff640 1 -- 172.21.15.23:0/3073843671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc009560 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.369+0000 7f33eb7fe640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33ebfff640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f33dc02c6a0 tx=0x7f33dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0044e0 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec19f670 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33dc004ac0 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc03e450 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.370+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec19fb80 con 0x7f33ec102050 2024-03-16T20:33:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.371+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec0fd470 con 0x7f33ec102050 2024-03-16T20:33:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.373+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f33dc003680 con 0x7f33ec102050 2024-03-16T20:33:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.373+0000 7f33e97fa640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 0x7f33d40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.373+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f33dc0ef9a0 con 0x7f33ec102050 2024-03-16T20:33:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.373+0000 7f33eb7fe640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 0x7f33d40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:38.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.374+0000 7f33eb7fe640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 0x7f33d40a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f33ec198ae0 tx=0x7f33e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:38.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.378+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f33dc0b9050 con 0x7f33ec102050 2024-03-16T20:33:38.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:38 smithi031 ceph-mon[36281]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:38 smithi189 ceph-mon[36655]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:38.538 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 systemd[1]: Starting Ceph mds.g for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:38.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.607+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33ec02cf90 con 0x7f33ec102050 2024-03-16T20:33:38.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.607+0000 7f33e97fa640 1 -- 172.21.15.23:0/3073843671 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7661 (secure 0 0 0) 0x7f33dc0bc710 con 0x7f33ec102050 2024-03-16T20:33:38.613 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:38.613 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-16T20:33:38.613 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 msgr2=0x7f33d40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 0x7f33d40a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f33ec198ae0 tx=0x7f33e0005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 msgr2=0x7f33ec1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f33dc02c6a0 tx=0x7f33dc0042f0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 shutdown_connections 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f33d40a1a30 0x7f33d40a3ef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33ec1072b0 0x7f33ec19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33ec104980 0x7f33ec197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 --2- 172.21.15.23:0/3073843671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33ec102050 0x7f33ec1975d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:38.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 >> 172.21.15.23:0/3073843671 conn(0x7f33ec0fb980 msgr2=0x7f33ec102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:38.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 shutdown_connections 2024-03-16T20:33:38.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:38.609+0000 7f33f258c640 1 -- 172.21.15.23:0/3073843671 wait complete. 2024-03-16T20:33:38.801 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 podman[70419]: 2024-03-16 20:33:38.436673715 +0000 UTC m=+0.018195924 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:38.802 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 podman[70419]: 2024-03-16 20:33:38.555633661 +0000 UTC m=+0.137155859 container create 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:33:38.802 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 podman[70419]: 2024-03-16 20:33:38.797233904 +0000 UTC m=+0.378756103 container init 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:33:39.052 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 podman[70419]: 2024-03-16 20:33:38.799886643 +0000 UTC m=+0.381408843 container start 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:33:39.052 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 ceph-mds[70440]: -- 172.21.15.23:0/488020577 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f7b65bc1a0 con 0x55f7b6593000 2024-03-16T20:33:39.052 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: starting mds.g at 2024-03-16T20:33:39.053 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 bash[70419]: 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 2024-03-16T20:33:39.053 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:33:38 smithi023 systemd[1]: Started Ceph mds.g for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:39.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3073843671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:39.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:39 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3073843671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:39.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:39 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:39.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3073843671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:39.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:39.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:39.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:39.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:39 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:40.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:40 smithi023 ceph-mon[31879]: Deploying daemon mds.a on smithi023 2024-03-16T20:33:40.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:40 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:boot 2024-03-16T20:33:40.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:40 smithi023 ceph-mon[31879]: fsmap 10 up:standby 2024-03-16T20:33:40.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:40 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:33:40.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:40 smithi023 ceph-mon[31879]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:40.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:40 smithi031 ceph-mon[36281]: Deploying daemon mds.a on smithi023 2024-03-16T20:33:40.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:40 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:boot 2024-03-16T20:33:40.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:40 smithi031 ceph-mon[36281]: fsmap 10 up:standby 2024-03-16T20:33:40.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:40 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:33:40.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:40 smithi031 ceph-mon[36281]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:40 smithi189 ceph-mon[36655]: Deploying daemon mds.a on smithi023 2024-03-16T20:33:40.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:40 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:boot 2024-03-16T20:33:40.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:40 smithi189 ceph-mon[36655]: fsmap 10 up:standby 2024-03-16T20:33:40.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:40 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:33:40.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:40 smithi189 ceph-mon[36655]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:40.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 -- 172.21.15.23:0/1848924374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 msgr2=0x7f1bfc104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 --2- 172.21.15.23:0/1848924374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc104380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009900 tx=0x7f1bec02bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 -- 172.21.15.23:0/1848924374 shutdown_connections 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 --2- 172.21.15.23:0/1848924374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 0x7f1bfc109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 --2- 172.21.15.23:0/1848924374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 0x7f1bfc106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 --2- 172.21.15.23:0/1848924374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc104380 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 -- 172.21.15.23:0/1848924374 >> 172.21.15.23:0/1848924374 conn(0x7f1bfc0fb980 msgr2=0x7f1bfc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 -- 172.21.15.23:0/1848924374 shutdown_connections 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.730+0000 7f1c0151e640 1 -- 172.21.15.23:0/1848924374 wait complete. 2024-03-16T20:33:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 Processor -- start 2024-03-16T20:33:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 -- start start 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 0x7f1bfc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 0x7f1bfc19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc19f070 con 0x7f1bfc101f90 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc19f1e0 con 0x7f1bfc1048c0 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1c0151e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc19f3a0 con 0x7f1bfc1071f0 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33194/0 (socket says 172.21.15.23:33194) 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 -- 172.21.15.23:0/3762489289 learned_addr learned my addr 172.21.15.23:0/3762489289 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfa7fc640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 0x7f1bfc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfb7fe640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 0x7f1bfc19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 -- 172.21.15.23:0/3762489289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 msgr2=0x7f1bfc19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 0x7f1bfc19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 -- 172.21.15.23:0/3762489289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 msgr2=0x7f1bfc197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 0x7f1bfc197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.731+0000 7f1bfaffd640 1 -- 172.21.15.23:0/3762489289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec009560 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1bfaffd640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1bec006010 tx=0x7f1bec005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0043d0 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bec0052d0 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec003cc0 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc19f620 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.732+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc19fb80 con 0x7f1bfc101f90 2024-03-16T20:33:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.733+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0fd470 con 0x7f1bfc101f90 2024-03-16T20:33:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.734+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1bec005440 con 0x7f1bfc101f90 2024-03-16T20:33:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.735+0000 7f1be3fff640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 0x7f1be40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.735+0000 7f1bfa7fc640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 0x7f1be40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.735+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1bec0ef6f0 con 0x7f1bfc101f90 2024-03-16T20:33:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.735+0000 7f1bfa7fc640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 0x7f1be40a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc198a90 tx=0x7f1bf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.742+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1bec0b8da0 con 0x7f1bfc101f90 2024-03-16T20:33:40.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.983+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bfc10cf20 con 0x7f1bfc101f90 2024-03-16T20:33:40.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.984+0000 7f1be3fff640 1 -- 172.21.15.23:0/3762489289 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8453 (secure 0 0 0) 0x7f1bec0bc460 con 0x7f1bfc101f90 2024-03-16T20:33:40.990 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:40.990 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-16T20:33:40.990 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 msgr2=0x7f1be40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 0x7f1be40a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc198a90 tx=0x7f1bf0005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 msgr2=0x7f1bfc197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1bec006010 tx=0x7f1bec005e80 comp rx=0 tx=0).stop 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 shutdown_connections 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1be40a1ab0 0x7f1be40a3f70 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc198a90 tx=0x7f1bf0005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bfc1071f0 0x7f1bfc19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bfc1048c0 0x7f1bfc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 --2- 172.21.15.23:0/3762489289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101f90 0x7f1bfc197580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:40.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.988+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 >> 172.21.15.23:0/3762489289 conn(0x7f1bfc0fb980 msgr2=0x7f1bfc102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:40.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.991+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 shutdown_connections 2024-03-16T20:33:40.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:40.991+0000 7f1c0151e640 1 -- 172.21.15.23:0/3762489289 wait complete. 2024-03-16T20:33:41.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:41 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3762489289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:41.341 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 systemd[1]: Starting Ceph mds.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:33:41.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:41 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3762489289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:41.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:41 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3762489289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:41.648 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 podman[70777]: 2024-03-16 20:33:41.273787111 +0000 UTC m=+0.018069148 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:33:41.649 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 podman[70777]: 2024-03-16 20:33:41.390345521 +0000 UTC m=+0.134627578 container create 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-16T20:33:41.649 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 podman[70777]: 2024-03-16 20:33:41.617831341 +0000 UTC m=+0.362113387 container init 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True) 2024-03-16T20:33:41.649 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 podman[70777]: 2024-03-16 20:33:41.620472947 +0000 UTC m=+0.364754977 container start 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T20:33:41.649 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 ceph-mds[70796]: -- 172.21.15.23:0/819504002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559584ff61a0 con 0x559584fcd000 2024-03-16T20:33:42.091 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: starting mds.a at 2024-03-16T20:33:42.091 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 bash[70777]: 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 2024-03-16T20:33:42.091 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:33:41 smithi023 systemd[1]: Started Ceph mds.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:43.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:43.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:43.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: Deploying daemon mds.b on smithi031 2024-03-16T20:33:43.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:43.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6840/4113023748,v1:172.21.15.23:6841/4113023748] up:boot 2024-03-16T20:33:43.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: fsmap 11 up:standby 2024-03-16T20:33:43.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:42 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:33:43.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 -- 172.21.15.23:0/3440164256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 msgr2=0x7f5bf8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/3440164256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8100b40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009900 tx=0x7f5be802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 -- 172.21.15.23:0/3440164256 shutdown_connections 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/3440164256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf8101080 0x7f5bf8101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/3440164256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8100b40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/3440164256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bf80ffd80 0x7f5bf81001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 -- 172.21.15.23:0/3440164256 >> 172.21.15.23:0/3440164256 conn(0x7f5bf80fb980 msgr2=0x7f5bf80fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 -- 172.21.15.23:0/3440164256 shutdown_connections 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.125+0000 7f5bffb5c640 1 -- 172.21.15.23:0/3440164256 wait complete. 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 Processor -- start 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 -- start start 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf80ffd80 0x7f5bf8199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:43.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bf8101080 0x7f5bf81a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf81a1240 con 0x7f5bf81006e0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf81a13b0 con 0x7f5bf80ffd80 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bffb5c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf81a1570 con 0x7f5bf8101080 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33240/0 (socket says 172.21.15.23:33240) 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 -- 172.21.15.23:0/249371212 learned_addr learned my addr 172.21.15.23:0/249371212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 -- 172.21.15.23:0/249371212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bf8101080 msgr2=0x7f5bf81a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd8f8640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf80ffd80 0x7f5bf8199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bf8101080 0x7f5bf81a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 -- 172.21.15.23:0/249371212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf80ffd80 msgr2=0x7f5bf8199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf80ffd80 0x7f5bf8199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 -- 172.21.15.23:0/249371212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8009560 con 0x7f5bf81006e0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.126+0000 7f5bfd0f7640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5be8006010 tx=0x7f5be80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.127+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be80044e0 con 0x7f5bf81006e0 2024-03-16T20:33:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.127+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf81a17f0 con 0x7f5bf81006e0 2024-03-16T20:33:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.127+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5be80054e0 con 0x7f5bf81006e0 2024-03-16T20:33:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.127+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be803e450 con 0x7f5bf81006e0 2024-03-16T20:33:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.127+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf81a1d50 con 0x7f5bf81006e0 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.128+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf810ac60 con 0x7f5bf81006e0 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.129+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5be80035f0 con 0x7f5bf81006e0 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.130+0000 7f5beeffd640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 0x7f5be40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.130+0000 7f5bfd8f8640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 0x7f5be40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.130+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5be80ef3c0 con 0x7f5bf81006e0 2024-03-16T20:33:43.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.130+0000 7f5bfd8f8640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 0x7f5be40a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005c40 tx=0x7f5bf4005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:43.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.135+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5be80b8a70 con 0x7f5bf81006e0 2024-03-16T20:33:43.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: Deploying daemon mds.b on smithi031 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6840/4113023748,v1:172.21.15.23:6841/4113023748] up:boot 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: fsmap 11 up:standby 2024-03-16T20:33:43.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:42 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:33:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: Deploying daemon mds.b on smithi031 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6840/4113023748,v1:172.21.15.23:6841/4113023748] up:boot 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: fsmap 11 up:standby 2024-03-16T20:33:43.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:42 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:33:43.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.353+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bf810f300 con 0x7f5bf81006e0 2024-03-16T20:33:43.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.353+0000 7f5beeffd640 1 -- 172.21.15.23:0/249371212 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9245 (secure 0 0 0) 0x7f5be80bc130 con 0x7f5bf81006e0 2024-03-16T20:33:43.359 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:43.359 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-16T20:33:43.359 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2024-03-16T20:33:43.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 msgr2=0x7f5be40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:43.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 0x7f5be40a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005c40 tx=0x7f5bf4005bd0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 msgr2=0x7f5bf8199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:43.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5be8006010 tx=0x7f5be80042f0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 shutdown_connections 2024-03-16T20:33:43.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5be40a1a30 0x7f5be40a3ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bf8101080 0x7f5bf81a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bf81006e0 0x7f5bf8199c90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 --2- 172.21.15.23:0/249371212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bf80ffd80 0x7f5bf8199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:43.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 >> 172.21.15.23:0/249371212 conn(0x7f5bf80fb980 msgr2=0x7f5bf80fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:43.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 shutdown_connections 2024-03-16T20:33:43.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:43.355+0000 7f5bffb5c640 1 -- 172.21.15.23:0/249371212 wait complete. 2024-03-16T20:33:44.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:43 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/249371212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:44.134 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:43 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/249371212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:44.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:43 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/249371212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:44.439 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 podman[66819]: 2024-03-16 20:33:44.359821425 +0000 UTC m=+0.125833548 container create 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 podman[66819]: 2024-03-16 20:33:44.620657693 +0000 UTC m=+0.386669784 container init 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 podman[66819]: 2024-03-16 20:33:44.62334645 +0000 UTC m=+0.389358538 container start 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 ceph-mds[66838]: -- 172.21.15.31:0/3543444139 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e26d5ca1a0 con 0x55e26d5a0c00 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: starting mds.b at 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 bash[66819]: 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 2024-03-16T20:33:44.812 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:33:44 smithi031 systemd[1]: Started Ceph mds.b for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:33:45.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:44 smithi023 ceph-mon[31879]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:45.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:44 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:44 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:44 smithi031 ceph-mon[36281]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:45.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:44 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:44 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:44 smithi189 ceph-mon[36655]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:45.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:44 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:44 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:45.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1407262833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 msgr2=0x7fe9e8101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1407262833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e8101210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009900 tx=0x7fe9d002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1407262833 shutdown_connections 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1407262833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 0x7fe9e8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1407262833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e8101210 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1407262833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 0x7fe9e80fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1407262833 >> 172.21.15.23:0/1407262833 conn(0x7fe9e80fb110 msgr2=0x7fe9e80fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1407262833 shutdown_connections 2024-03-16T20:33:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.472+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1407262833 wait complete. 2024-03-16T20:33:45.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 Processor -- start 2024-03-16T20:33:45.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 -- start start 2024-03-16T20:33:45.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 0x7fe9e8109470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 0x7fe9e819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e819f080 con 0x7fe9e8100d90 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e819f1f0 con 0x7fe9e80ffba0 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9ec95c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e819f3b0 con 0x7fe9e8101750 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33260/0 (socket says 172.21.15.23:33260) 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 -- 172.21.15.23:0/1639787868 learned_addr learned my addr 172.21.15.23:0/1639787868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e6d9d640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 0x7fe9e819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e659c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 0x7fe9e8109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 -- 172.21.15.23:0/1639787868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 msgr2=0x7fe9e819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 0x7fe9e819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 -- 172.21.15.23:0/1639787868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 msgr2=0x7fe9e8109470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 0x7fe9e8109470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 -- 172.21.15.23:0/1639787868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0009560 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.473+0000 7fe9e5d9b640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0005e40 tx=0x7fe9d0005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.474+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0004250 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.474+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d00043b0 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.474+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e819f630 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.474+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0003bc0 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.474+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e819fb90 con 0x7fe9e8100d90 2024-03-16T20:33:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.475+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e8106e60 con 0x7fe9e8100d90 2024-03-16T20:33:45.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.476+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe9d0003d20 con 0x7fe9e8100d90 2024-03-16T20:33:45.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.476+0000 7fe9df7fe640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 0x7fe9d40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.477+0000 7fe9e659c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 0x7fe9d40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.477+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe9d00ef0b0 con 0x7fe9e8100d90 2024-03-16T20:33:45.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.477+0000 7fe9e659c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 0x7fe9d40a3f70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8100bf0 tx=0x7fe9e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:45.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.481+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe9d00b8760 con 0x7fe9e8100d90 2024-03-16T20:33:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.730+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e810c000 con 0x7fe9e8100d90 2024-03-16T20:33:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.731+0000 7fe9df7fe640 1 -- 172.21.15.23:0/1639787868 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10031 (secure 0 0 0) 0x7fe9d00bbe20 con 0x7fe9e8100d90 2024-03-16T20:33:45.737 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:45.737 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24497,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-16T20:33:45.737 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 msgr2=0x7fe9d40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 0x7fe9d40a3f70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8100bf0 tx=0x7fe9e0005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 msgr2=0x7fe9e819c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0005e40 tx=0x7fe9d0005e70 comp rx=0 tx=0).stop 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 shutdown_connections 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9d40a1ab0 0x7fe9d40a3f70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e8101750 0x7fe9e819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9e8100d90 0x7fe9e819c2b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 --2- 172.21.15.23:0/1639787868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9e80ffba0 0x7fe9e8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.733+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 >> 172.21.15.23:0/1639787868 conn(0x7fe9e80fb110 msgr2=0x7fe9e8104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.734+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 shutdown_connections 2024-03-16T20:33:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.734+0000 7fe9ec95c640 1 -- 172.21.15.23:0/1639787868 wait complete. 2024-03-16T20:33:45.750 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 -- 172.21.15.23:0/390444977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c101fb0 msgr2=0x7fc13c1043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 --2- 172.21.15.23:0/390444977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c101fb0 0x7fc13c1043a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc138009900 tx=0x7fc13802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 -- 172.21.15.23:0/390444977 shutdown_connections 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 --2- 172.21.15.23:0/390444977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c107210 0x7fc13c109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 --2- 172.21.15.23:0/390444977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 --2- 172.21.15.23:0/390444977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c101fb0 0x7fc13c1043a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 -- 172.21.15.23:0/390444977 >> 172.21.15.23:0/390444977 conn(0x7fc13c0fb9a0 msgr2=0x7fc13c0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 -- 172.21.15.23:0/390444977 shutdown_connections 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.847+0000 7fc1435a1640 1 -- 172.21.15.23:0/390444977 wait complete. 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 Processor -- start 2024-03-16T20:33:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 -- start start 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c101fb0 0x7fc13c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c19f0c0 con 0x7fc13c107210 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c19f230 con 0x7fc13c101fb0 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc1435a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c19f3f0 con 0x7fc13c1048e0 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33278/0 (socket says 172.21.15.23:33278) 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 -- 172.21.15.23:0/4058555155 learned_addr learned my addr 172.21.15.23:0/4058555155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc14133d640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c101fb0 0x7fc13c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc140b3c640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 -- 172.21.15.23:0/4058555155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 msgr2=0x7fc13c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 -- 172.21.15.23:0/4058555155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c101fb0 msgr2=0x7fc13c1975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c101fb0 0x7fc13c1975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 -- 172.21.15.23:0/4058555155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138009560 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc140b3c640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.848+0000 7fc141b3e640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc13400eb10 tx=0x7fc13400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.849+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134004b20 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.849+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc134004c80 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.849+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c19f6d0 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.849+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134010650 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.849+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c19fbe0 con 0x7fc13c107210 2024-03-16T20:33:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.850+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c0fd490 con 0x7fc13c107210 2024-03-16T20:33:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.851+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc134005020 con 0x7fc13c107210 2024-03-16T20:33:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.851+0000 7fc1327fc640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 0x7fc1280a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.851+0000 7fc14133d640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 0x7fc1280a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.851+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc134014070 con 0x7fc13c107210 2024-03-16T20:33:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.852+0000 7fc14133d640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 0x7fc1280a3f70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc138009230 tx=0x7fc138005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:45.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:45.856+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc134096b60 con 0x7fc13c107210 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:boot 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: fsmap 12 up:standby 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:45 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1639787868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.042+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc13c02d000 con 0x7fc13c107210 2024-03-16T20:33:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.043+0000 7fc1327fc640 1 -- 172.21.15.23:0/4058555155 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18410 (secure 0 0 0) 0x7fc13409a220 con 0x7fc13c107210 2024-03-16T20:33:46.049 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:46.049 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:32:59.716148+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 msgr2=0x7fc1280a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 0x7fc1280a3f70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc138009230 tx=0x7fc138005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 msgr2=0x7fc13c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc13400eb10 tx=0x7fc13400c2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 shutdown_connections 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc1280a1ab0 0x7fc1280a3f70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc13c107210 0x7fc13c19eb80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc13c1048e0 0x7fc13c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 --2- 172.21.15.23:0/4058555155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc13c101fb0 0x7fc13c1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 >> 172.21.15.23:0/4058555155 conn(0x7fc13c0fb9a0 msgr2=0x7fc13c102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 shutdown_connections 2024-03-16T20:33:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.045+0000 7fc1435a1640 1 -- 172.21.15.23:0/4058555155 wait complete. 2024-03-16T20:33:46.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T20:33:46.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:boot 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: fsmap 12 up:standby 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.129 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.130 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:45 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1639787868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 -- 172.21.15.23:0/3073756950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 msgr2=0x7f73d0101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/3073756950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d0101a60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009900 tx=0x7f73c002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 -- 172.21.15.23:0/3073756950 shutdown_connections 2024-03-16T20:33:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/3073756950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/3073756950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d0101a60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/3073756950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 0x7f73d0100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 -- 172.21.15.23:0/3073756950 >> 172.21.15.23:0/3073756950 conn(0x7f73d00fb980 msgr2=0x7f73d00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 -- 172.21.15.23:0/3073756950 shutdown_connections 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 -- 172.21.15.23:0/3073756950 wait complete. 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.164+0000 7f73d7ab5640 1 Processor -- start 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 -- start start 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 0x7f73d010acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0109f90 con 0x7f73d01015e0 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d010a100 con 0x7f73d0101fa0 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d7ab5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d01a1600 con 0x7f73d01003f0 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d6052640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33290/0 (socket says 172.21.15.23:33290) 2024-03-16T20:33:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d6052640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:48916/0 (socket says 172.21.15.23:48916) 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 -- 172.21.15.23:0/2619699865 learned_addr learned my addr 172.21.15.23:0/2619699865 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5851640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 0x7f73d010acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 -- 172.21.15.23:0/2619699865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 msgr2=0x7f73d010acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 0x7f73d010acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 -- 172.21.15.23:0/2619699865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 msgr2=0x7f73d0109830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 -- 172.21.15.23:0/2619699865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0009560 con 0x7f73d01015e0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d6052640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73d5050640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73c0036040 tx=0x7f73c0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.165+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c00043d0 con 0x7f73d01015e0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.166+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d01a1850 con 0x7f73d01015e0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.166+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73c00052d0 con 0x7f73d01015e0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.166+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0003cc0 con 0x7f73d01015e0 2024-03-16T20:33:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.166+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01a1d60 con 0x7f73d01015e0 2024-03-16T20:33:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.166+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0107760 con 0x7f73d01015e0 2024-03-16T20:33:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.167+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f73c0005440 con 0x7f73d01015e0 2024-03-16T20:33:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.168+0000 7f73c6ffd640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 0x7f73bc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.168+0000 7f73d5851640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 0x7f73bc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.168+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f73c00ef700 con 0x7f73d01015e0 2024-03-16T20:33:46.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.168+0000 7f73d5851640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 0x7f73bc0a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f73d0101440 tx=0x7f73cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.171+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f73c00b8db0 con 0x7f73d01015e0 2024-03-16T20:33:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:boot 2024-03-16T20:33:46.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: fsmap 12 up:standby 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:45 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1639787868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.357+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73d010d150 con 0x7f73d01015e0 2024-03-16T20:33:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.358+0000 7f73c6ffd640 1 -- 172.21.15.23:0/2619699865 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73c00bc470 con 0x7f73d01015e0 2024-03-16T20:33:46.363 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:33:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 msgr2=0x7f73bc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 0x7f73bc0a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f73d0101440 tx=0x7f73cc005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 msgr2=0x7f73d01092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73c0036040 tx=0x7f73c0005e80 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 shutdown_connections 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73bc0a1a30 0x7f73bc0a3ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73d0101fa0 0x7f73d0109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73d01015e0 0x7f73d01092f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 --2- 172.21.15.23:0/2619699865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73d01003f0 0x7f73d010acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 >> 172.21.15.23:0/2619699865 conn(0x7f73d00fb980 msgr2=0x7f73d0104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 shutdown_connections 2024-03-16T20:33:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.361+0000 7f73d7ab5640 1 -- 172.21.15.23:0/2619699865 wait complete. 2024-03-16T20:33:46.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- 172.21.15.23:0/3867439612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 msgr2=0x7f12dc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 --2- 172.21.15.23:0/3867439612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc100830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009900 tx=0x7f12d802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- 172.21.15.23:0/3867439612 shutdown_connections 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 --2- 172.21.15.23:0/3867439612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 0x7f12dc108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 --2- 172.21.15.23:0/3867439612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 0x7f12dc101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 --2- 172.21.15.23:0/3867439612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc100830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- 172.21.15.23:0/3867439612 >> 172.21.15.23:0/3867439612 conn(0x7f12dc0fb980 msgr2=0x7f12dc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- 172.21.15.23:0/3867439612 shutdown_connections 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- 172.21.15.23:0/3867439612 wait complete. 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 Processor -- start 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.484+0000 7f12e395c640 1 -- start start 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 0x7f12dc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 0x7f12dc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc19f0b0 con 0x7f12dc100410 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc19f220 con 0x7f12dc101fc0 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc19f3e0 con 0x7f12dc101600 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33308/0 (socket says 172.21.15.23:33308) 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 -- 172.21.15.23:0/1634266117 learned_addr learned my addr 172.21.15.23:0/1634266117 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e1ef9640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 0x7f12dc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e0ef7640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 0x7f12dc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 -- 172.21.15.23:0/1634266117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 msgr2=0x7f12dc197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 0x7f12dc197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 -- 172.21.15.23:0/1634266117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 msgr2=0x7f12dc19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 0x7f12dc19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 -- 172.21.15.23:0/1634266117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d8009560 con 0x7f12dc100410 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e16f8640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005ab0 tx=0x7f12d8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d80043d0 con 0x7f12dc100410 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12dc19f660 con 0x7f12dc100410 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d80052d0 con 0x7f12dc100410 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8003d40 con 0x7f12dc100410 2024-03-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.485+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12dc19fbc0 con 0x7f12dc100410 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.487+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12dc10be20 con 0x7f12dc100410 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.488+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f12d8005440 con 0x7f12dc100410 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.488+0000 7f12d27fc640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 0x7f12c80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.488+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12d80ef130 con 0x7f12dc100410 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.488+0000 7f12e0ef7640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 0x7f12c80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.489+0000 7f12e0ef7640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 0x7f12c80a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f12dc198ad0 tx=0x7f12cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.493+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f12d80b87e0 con 0x7f12dc100410 2024-03-16T20:33:46.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.746+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f12dc10bfb0 con 0x7f12dc100410 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.746+0000 7f12d27fc640 1 -- 172.21.15.23:0/1634266117 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f12d80bbea0 con 0x7f12dc100410 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:33:46.752 INFO:teuthology.orchestra.run.smithi023.stdout:] 2024-03-16T20:33:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 msgr2=0x7f12c80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 0x7f12c80a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f12dc198ad0 tx=0x7f12cc005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 msgr2=0x7f12dc1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005ab0 tx=0x7f12d8005e50 comp rx=0 tx=0).stop 2024-03-16T20:33:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 shutdown_connections 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f12c80a1a30 0x7f12c80a3ef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12dc101fc0 0x7f12dc19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc101600 0x7f12dc197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 --2- 172.21.15.23:0/1634266117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12dc100410 0x7f12dc1975c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 >> 172.21.15.23:0/1634266117 conn(0x7f12dc0fb980 msgr2=0x7f12dc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 shutdown_connections 2024-03-16T20:33:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.750+0000 7f12e395c640 1 -- 172.21.15.23:0/1634266117 wait complete. 2024-03-16T20:33:46.772 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-16T20:33:46.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 -- 172.21.15.23:0/126579938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f101026f0 msgr2=0x7f3f10102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 --2- 172.21.15.23:0/126579938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f101026f0 0x7f3f10102b70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009900 tx=0x7f3f0402bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 -- 172.21.15.23:0/126579938 shutdown_connections 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 --2- 172.21.15.23:0/126579938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101030b0 0x7f3f10109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 --2- 172.21.15.23:0/126579938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f101026f0 0x7f3f10102b70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 --2- 172.21.15.23:0/126579938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f10101500 0x7f3f10101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 -- 172.21.15.23:0/126579938 >> 172.21.15.23:0/126579938 conn(0x7f3f100fca90 msgr2=0x7f3f100feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 -- 172.21.15.23:0/126579938 shutdown_connections 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.874+0000 7f3f170b5640 1 -- 172.21.15.23:0/126579938 wait complete. 2024-03-16T20:33:46.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 Processor -- start 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 -- start start 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101026f0 0x7f3f10197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f101030b0 0x7f3f1019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1019f080 con 0x7f3f10101500 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1019f1f0 con 0x7f3f101030b0 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f170b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1019f3b0 con 0x7f3f101026f0 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33326/0 (socket says 172.21.15.23:33326) 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 -- 172.21.15.23:0/2223653675 learned_addr learned my addr 172.21.15.23:0/2223653675 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f15652640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f101030b0 0x7f3f1019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f0ffff640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101026f0 0x7f3f10197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 -- 172.21.15.23:0/2223653675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101026f0 msgr2=0x7f3f10197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101026f0 0x7f3f10197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 -- 172.21.15.23:0/2223653675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f101030b0 msgr2=0x7f3f1019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f101030b0 0x7f3f1019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.875+0000 7f3f14e51640 1 -- 172.21.15.23:0/2223653675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04009560 con 0x7f3f10101500 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f14e51640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800ebc0 tx=0x7f3f0800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08003b70 con 0x7f3f10101500 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f0800eec0 con 0x7f3f10101500 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1019f690 con 0x7f3f10101500 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08019800 con 0x7f3f10101500 2024-03-16T20:33:46.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.876+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1019fbc0 con 0x7f3f10101500 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.877+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f10068730 con 0x7f3f10101500 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.878+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3f08019960 con 0x7f3f10101500 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.878+0000 7f3f0dffb640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 0x7f3ef80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.878+0000 7f3f0ffff640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 0x7f3ef80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.878+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f08014070 con 0x7f3f10101500 2024-03-16T20:33:46.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.879+0000 7f3f0ffff640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 0x7f3ef80a3f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009230 tx=0x7f3f04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:46.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:46.883+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3f08096dd0 con 0x7f3f10101500 2024-03-16T20:33:47.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.065+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3f101088a0 con 0x7f3f10101500 2024-03-16T20:33:47.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:47.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4058555155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:47.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2619699865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:47.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1634266117' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T20:33:47.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:46 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4058555155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2619699865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1634266117' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:46 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4058555155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2619699865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:33:47.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1634266117' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-16T20:33:47.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:46 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:47.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.824+0000 7f3f0dffb640 1 -- 172.21.15.23:0/2223653675 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3f0809a490 con 0x7f3f10101500 2024-03-16T20:33:47.829 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2024-03-16T20:33:47.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 msgr2=0x7f3ef80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:47.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 0x7f3ef80a3f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009230 tx=0x7f3f04005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 msgr2=0x7f3f10197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800ebc0 tx=0x7f3f0800c6a0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 shutdown_connections 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ef80a1ab0 0x7f3ef80a3f70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f101030b0 0x7f3f1019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f101026f0 0x7f3f10197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 --2- 172.21.15.23:0/2223653675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f10101500 0x7f3f10197590 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 >> 172.21.15.23:0/2223653675 conn(0x7f3f100fca90 msgr2=0x7f3f10105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 shutdown_connections 2024-03-16T20:33:47.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.826+0000 7f3f170b5640 1 -- 172.21.15.23:0/2223653675 wait complete. 2024-03-16T20:33:47.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- 172.21.15.23:0/1784698365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec100430 msgr2=0x7f13ec100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1784698365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec100430 0x7f13ec100850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009900 tx=0x7f13e802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- 172.21.15.23:0/1784698365 shutdown_connections 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1784698365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101fe0 0x7f13ec108870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1784698365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec101620 0x7f13ec101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1784698365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec100430 0x7f13ec100850 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- 172.21.15.23:0/1784698365 >> 172.21.15.23:0/1784698365 conn(0x7f13ec0fb9c0 msgr2=0x7f13ec0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- 172.21.15.23:0/1784698365 shutdown_connections 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- 172.21.15.23:0/1784698365 wait complete. 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 Processor -- start 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.955+0000 7f13f3f48640 1 -- start start 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 0x7f13ec06a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101620 0x7f13ec19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec19f120 con 0x7f13ec101fe0 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec19f290 con 0x7f13ec101620 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec19f450 con 0x7f13ec100430 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f1ce4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 0x7f13ec06a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33340/0 (socket says 172.21.15.23:33340) 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f14e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101620 0x7f13ec19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f1ce4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 0x7f13ec06a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:48792/0 (socket says 172.21.15.23:48792) 2024-03-16T20:33:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 -- 172.21.15.23:0/1008309648 learned_addr learned my addr 172.21.15.23:0/1008309648 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 -- 172.21.15.23:0/1008309648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 msgr2=0x7f13ec06a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 0x7f13ec06a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 -- 172.21.15.23:0/1008309648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101620 msgr2=0x7f13ec19c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101620 0x7f13ec19c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 -- 172.21.15.23:0/1008309648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8009560 con 0x7f13ec101fe0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f24e5640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f13e400eb10 tx=0x7f13e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4004b20 con 0x7f13ec101fe0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec19f6d0 con 0x7f13ec101fe0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13e4004c80 con 0x7f13ec101fe0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4010650 con 0x7f13ec101fe0 2024-03-16T20:33:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.956+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec19fb90 con 0x7f13ec101fe0 2024-03-16T20:33:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.958+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec107750 con 0x7f13ec101fe0 2024-03-16T20:33:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.959+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f13e40107f0 con 0x7f13ec101fe0 2024-03-16T20:33:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.960+0000 7f13e2ffd640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 0x7f13d80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.960+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f13e4014070 con 0x7f13ec101fe0 2024-03-16T20:33:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.960+0000 7f13f1ce4640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 0x7f13d80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.961+0000 7f13f1ce4640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 0x7f13d80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13e8005ab0 tx=0x7f13e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:47.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:47.964+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f13e4096d70 con 0x7f13ec101fe0 2024-03-16T20:33:48.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T20:33:48.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:48.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:48.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:47 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:48.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.154+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f13ec10be40 con 0x7f13ec101fe0 2024-03-16T20:33:48.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T20:33:48.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:48.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:48.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:47 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:48.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:47 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:33:48.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.830+0000 7f13e2ffd640 1 -- 172.21.15.23:0/1008309648 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) v1 ==== 171+0+0 (secure 0 0 0) 0x7f13e409a430 con 0x7f13ec101fe0 2024-03-16T20:33:48.835 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2024-03-16T20:33:48.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.833+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 msgr2=0x7f13d80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.833+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 0x7f13d80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f13e8005ab0 tx=0x7f13e8005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 msgr2=0x7f13ec19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f13e400eb10 tx=0x7f13e400c2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 shutdown_connections 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f13d80a1a30 0x7f13d80a3ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13ec101fe0 0x7f13ec19e940 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13ec101620 0x7f13ec19c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 --2- 172.21.15.23:0/1008309648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13ec100430 0x7f13ec06a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 >> 172.21.15.23:0/1008309648 conn(0x7f13ec0fb9c0 msgr2=0x7f13ec104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 shutdown_connections 2024-03-16T20:33:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.834+0000 7f13f3f48640 1 -- 172.21.15.23:0/1008309648 wait complete. 2024-03-16T20:33:48.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-16T20:33:48.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2879922083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 msgr2=0x7ff4d8100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2879922083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8100810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009900 tx=0x7ff4d402bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2879922083 shutdown_connections 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2879922083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d8101fa0 0x7ff4d8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2879922083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d81015e0 0x7ff4d8101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.974+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2879922083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8100810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2879922083 >> 172.21.15.23:0/2879922083 conn(0x7ff4d80fb980 msgr2=0x7ff4d80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2879922083 shutdown_connections 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2879922083 wait complete. 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 Processor -- start 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- start start 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81015e0 0x7ff4d8196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d8101fa0 0x7ff4d819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:48.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d819e020 con 0x7ff4d81003f0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d819e190 con 0x7ff4d8101fa0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4e05c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d819e350 con 0x7ff4d81015e0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4de363640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4de363640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33354/0 (socket says 172.21.15.23:33354) 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.975+0000 7ff4de363640 1 -- 172.21.15.23:0/2384023991 learned_addr learned my addr 172.21.15.23:0/2384023991 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4deb64640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d8101fa0 0x7ff4d819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4ddb62640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81015e0 0x7ff4d8196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 -- 172.21.15.23:0/2384023991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81015e0 msgr2=0x7ff4d8196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81015e0 0x7ff4d8196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 -- 172.21.15.23:0/2384023991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d8101fa0 msgr2=0x7ff4d819dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d8101fa0 0x7ff4d819dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 -- 172.21.15.23:0/2384023991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4009560 con 0x7ff4d81003f0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4de363640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40044c0 con 0x7ff4d81003f0 2024-03-16T20:33:48.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d819e5d0 con 0x7ff4d81003f0 2024-03-16T20:33:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d40052d0 con 0x7ff4d81003f0 2024-03-16T20:33:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4003e80 con 0x7ff4d81003f0 2024-03-16T20:33:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.976+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d819eb30 con 0x7ff4d81003f0 2024-03-16T20:33:48.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.977+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8107830 con 0x7ff4d81003f0 2024-03-16T20:33:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.979+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff4d4005440 con 0x7ff4d81003f0 2024-03-16T20:33:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.980+0000 7ff4cf7fe640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 0x7ff4c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.980+0000 7ff4ddb62640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 0x7ff4c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.980+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7ff4d40efec0 con 0x7ff4d81003f0 2024-03-16T20:33:48.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.980+0000 7ff4ddb62640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 0x7ff4c40a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8197a40 tx=0x7ff4c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:48.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:48.984+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff4d40b9280 con 0x7ff4d81003f0 2024-03-16T20:33:49.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:48 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T20:33:49.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:48 smithi023 ceph-mon[31879]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T20:33:49.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:48 smithi023 ceph-mon[31879]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:49.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:48 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T20:33:49.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:48 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T20:33:49.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:48 smithi031 ceph-mon[36281]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T20:33:49.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:48 smithi031 ceph-mon[36281]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:49.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:48 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T20:33:49.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:48 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2223653675' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-16T20:33:49.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:48 smithi189 ceph-mon[36655]: osdmap e83: 12 total, 12 up, 12 in 2024-03-16T20:33:49.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:48 smithi189 ceph-mon[36655]: pgmap v282: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:49.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:48 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-16T20:33:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:49.237+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff4d810be20 con 0x7ff4d81003f0 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:50.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:50.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T20:33:50.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:49 smithi023 ceph-mon[31879]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-16T20:33:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T20:33:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T20:33:50.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T20:33:50.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:49 smithi031 ceph-mon[36281]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-16T20:33:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.208+0000 7ff4cf7fe640 1 -- 172.21.15.23:0/2384023991 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff4d40430c0 con 0x7ff4d81003f0 2024-03-16T20:33:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2024-03-16T20:33:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 msgr2=0x7ff4c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 0x7ff4c40a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8197a40 tx=0x7ff4c8005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 msgr2=0x7ff4d8196530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000c00 tx=0x7ff4d40040c0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 shutdown_connections 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4c40a1a30 0x7ff4c40a3ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4d8101fa0 0x7ff4d819dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81015e0 0x7ff4d8196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 --2- 172.21.15.23:0/2384023991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4d81003f0 0x7ff4d8196530 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 >> 172.21.15.23:0/2384023991 conn(0x7ff4d80fb980 msgr2=0x7ff4d8104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 shutdown_connections 2024-03-16T20:33:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.212+0000 7ff4e05c7640 1 -- 172.21.15.23:0/2384023991 wait complete. 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1008309648' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: osdmap e84: 12 total, 12 up, 12 in 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:33:50.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:33:50.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:33:50.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-16T20:33:50.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:49 smithi189 ceph-mon[36655]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-16T20:33:50.232 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-16T20:33:50.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- 172.21.15.23:0/2553443427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 msgr2=0x7f0f90108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 --2- 172.21.15.23:0/2553443427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f90108850 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009ca0 tx=0x7f0f8c02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- 172.21.15.23:0/2553443427 shutdown_connections 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 --2- 172.21.15.23:0/2553443427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f90108850 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 --2- 172.21.15.23:0/2553443427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90101600 0x7f0f90101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 --2- 172.21.15.23:0/2553443427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90100410 0x7f0f90100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- 172.21.15.23:0/2553443427 >> 172.21.15.23:0/2553443427 conn(0x7f0f900fb980 msgr2=0x7f0f900fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- 172.21.15.23:0/2553443427 shutdown_connections 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- 172.21.15.23:0/2553443427 wait complete. 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 Processor -- start 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.338+0000 7f0f98393640 1 -- start start 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90100410 0x7f0f901975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90101600 0x7f0f90197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9019f0a0 con 0x7f0f90101fc0 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9019f210 con 0x7f0f90100410 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9019f3d0 con 0x7f0f90101600 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33382/0 (socket says 172.21.15.23:33382) 2024-03-16T20:33:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 -- 172.21.15.23:0/2358600274 learned_addr learned my addr 172.21.15.23:0/2358600274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f9612f640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90100410 0x7f0f901975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f9592e640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90101600 0x7f0f90197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 -- 172.21.15.23:0/2358600274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90101600 msgr2=0x7f0f90197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90101600 0x7f0f90197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 -- 172.21.15.23:0/2358600274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90100410 msgr2=0x7f0f901975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90100410 0x7f0f901975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 -- 172.21.15.23:0/2358600274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c009900 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f96930640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c037040 tx=0x7f0f8c034d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c003970 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f8c004040 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9019f650 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.339+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0406b0 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.340+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9019fb00 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.340+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9010be80 con 0x7f0f90101fc0 2024-03-16T20:33:50.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.342+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0f8c00a040 con 0x7f0f90101fc0 2024-03-16T20:33:50.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.342+0000 7f0f877fe640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 0x7f0f7c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.342+0000 7f0f9612f640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 0x7f0f7c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.342+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0f8c0f0920 con 0x7f0f90101fc0 2024-03-16T20:33:50.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.342+0000 7f0f9612f640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 0x7f0f7c0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f90101460 tx=0x7f0f88009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:50.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.346+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0f8c0b9c80 con 0x7f0f90101fc0 2024-03-16T20:33:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.536+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f0f9010c010 con 0x7f0f90101fc0 2024-03-16T20:33:50.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.842+0000 7f0f877fe640 1 -- 172.21.15.23:0/2358600274 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) v1 ==== 157+0+0 (secure 0 0 0) 0x7f0f8c0bd340 con 0x7f0f90101fc0 2024-03-16T20:33:50.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.844+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 msgr2=0x7f0f7c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.844+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 0x7f0f7c0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f90101460 tx=0x7f0f88009110 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.844+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 msgr2=0x7f0f9019eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.844+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c037040 tx=0x7f0f8c034d20 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 shutdown_connections 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f7c0a1a30 0x7f0f7c0a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f90101fc0 0x7f0f9019eb60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f90101600 0x7f0f90197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 --2- 172.21.15.23:0/2358600274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f90100410 0x7f0f901975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 >> 172.21.15.23:0/2358600274 conn(0x7f0f900fb980 msgr2=0x7f0f90069b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 shutdown_connections 2024-03-16T20:33:50.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.845+0000 7f0f98393640 1 -- 172.21.15.23:0/2358600274 wait complete. 2024-03-16T20:33:50.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-16T20:33:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4238880033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c54101520 msgr2=0x7f9c54101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4238880033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c54101520 0x7f9c54101940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009900 tx=0x7f9c4802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4238880033 shutdown_connections 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4238880033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c541030d0 0x7f9c54109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4238880033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54102710 0x7f9c54102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4238880033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c54101520 0x7f9c54101940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4238880033 >> 172.21.15.23:0/4238880033 conn(0x7f9c540fca90 msgr2=0x7f9c540feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4238880033 shutdown_connections 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.967+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4238880033 wait complete. 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 Processor -- start 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 -- start start 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54101520 0x7f9c54197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 0x7f9c54197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5419f060 con 0x7f9c541030d0 2024-03-16T20:33:50.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5419f1d0 con 0x7f9c54102710 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c5abb6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5419f390 con 0x7f9c54101520 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46276/0 (socket says 172.21.15.23:46276) 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 -- 172.21.15.23:0/4276440795 learned_addr learned my addr 172.21.15.23:0/4276440795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c58952640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54101520 0x7f9c54197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 -- 172.21.15.23:0/4276440795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54101520 msgr2=0x7f9c54197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54101520 0x7f9c54197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 -- 172.21.15.23:0/4276440795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 msgr2=0x7f9c54197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c53fff640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 0x7f9c54197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 0x7f9c54197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c59153640 1 -- 172.21.15.23:0/4276440795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48009560 con 0x7f9c541030d0 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.968+0000 7f9c53fff640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 0x7f9c54197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c59153640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00cce0 tx=0x7f9c4c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c00f450 con 0x7f9c541030d0 2024-03-16T20:33:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c4c00fa90 con 0x7f9c541030d0 2024-03-16T20:33:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5419f670 con 0x7f9c541030d0 2024-03-16T20:33:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c018b90 con 0x7f9c541030d0 2024-03-16T20:33:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.969+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5419fb80 con 0x7f9c541030d0 2024-03-16T20:33:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.970+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c54108770 con 0x7f9c541030d0 2024-03-16T20:33:50.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.971+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9c4c00fc00 con 0x7f9c541030d0 2024-03-16T20:33:50.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.971+0000 7f9c51ffb640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 0x7f9c440a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:50.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.972+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9c4c0cdbe0 con 0x7f9c541030d0 2024-03-16T20:33:50.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.972+0000 7f9c58952640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 0x7f9c440a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:50.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.972+0000 7f9c58952640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 0x7f9c440a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4802c6a0 tx=0x7f9c48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:50.976+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9c4c096dc0 con 0x7f9c541030d0 2024-03-16T20:33:51.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:51.160+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9c5402d000 con 0x7f9c541030d0 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: pgmap v285: 97 pgs: 13 creating+peering, 75 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-16T20:33:51.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:50 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T20:33:51.210 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T20:33:51.210 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: pgmap v285: 97 pgs: 13 creating+peering, 75 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:51.210 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T20:33:51.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-16T20:33:51.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-16T20:33:51.211 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:50 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T20:33:51.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: osdmap e85: 12 total, 12 up, 12 in 2024-03-16T20:33:51.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: pgmap v285: 97 pgs: 13 creating+peering, 75 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-16T20:33:51.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2384023991' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-16T20:33:51.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-16T20:33:51.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-16T20:33:51.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:50 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:active 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:51 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:52.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:33:52.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T20:33:52.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T20:33:52.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T20:33:52.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:active 2024-03-16T20:33:52.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.190 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:51 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2358600274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: osdmap e86: 12 total, 12 up, 12 in 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] up:active 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:51 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:52.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.858+0000 7f9c51ffb640 1 -- 172.21.15.23:0/4276440795 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) v1 ==== 228+0+0 (secure 0 0 0) 0x7f9c4c09a480 con 0x7f9c541030d0 2024-03-16T20:33:52.864 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 msgr2=0x7f9c440a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 0x7f9c440a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4802c6a0 tx=0x7f9c48005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 msgr2=0x7f9c5419eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00cce0 tx=0x7f9c4c00a620 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 shutdown_connections 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9c440a1ab0 0x7f9c440a3f70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c541030d0 0x7f9c5419eb20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c54102710 0x7f9c54197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 --2- 172.21.15.23:0/4276440795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c54101520 0x7f9c54197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 >> 172.21.15.23:0/4276440795 conn(0x7f9c540fca90 msgr2=0x7f9c54105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 shutdown_connections 2024-03-16T20:33:52.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.861+0000 7f9c5abb6640 1 -- 172.21.15.23:0/4276440795 wait complete. 2024-03-16T20:33:52.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-16T20:33:53.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- 172.21.15.23:0/3813015049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc1048c0 msgr2=0x7f16cc106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:53.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/3813015049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc1048c0 0x7f16cc106cb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009900 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:53.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- 172.21.15.23:0/3813015049 shutdown_connections 2024-03-16T20:33:53.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/3813015049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1071f0 0x7f16cc109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/3813015049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc1048c0 0x7f16cc106cb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/3813015049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc101f90 0x7f16cc104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- 172.21.15.23:0/3813015049 >> 172.21.15.23:0/3813015049 conn(0x7f16cc0fb9a0 msgr2=0x7f16cc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- 172.21.15.23:0/3813015049 shutdown_connections 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- 172.21.15.23:0/3813015049 wait complete. 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 Processor -- start 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.991+0000 7f16d2b3a640 1 -- start start 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1048c0 0x7f16cc0fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc1071f0 0x7f16cc100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc100be0 con 0x7f16cc101f90 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc100d50 con 0x7f16cc1071f0 2024-03-16T20:33:53.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc1a1660 con 0x7f16cc1048c0 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46298/0 (socket says 172.21.15.23:46298) 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 -- 172.21.15.23:0/1217390323 learned_addr learned my addr 172.21.15.23:0/1217390323 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16cbfff640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1048c0 0x7f16cc0fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d10d7640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc1071f0 0x7f16cc100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 -- 172.21.15.23:0/1217390323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1048c0 msgr2=0x7f16cc0fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1048c0 0x7f16cc0fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 -- 172.21.15.23:0/1217390323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc1071f0 msgr2=0x7f16cc100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc1071f0 0x7f16cc100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 -- 172.21.15.23:0/1217390323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc009560 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d08d6640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f16c400eae0 tx=0x7f16c400c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c400ce50 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16c4011950 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16cc1a1910 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.992+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c401ab10 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.993+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16cc1a1df0 con 0x7f16cc101f90 2024-03-16T20:33:53.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.993+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16cc0fd490 con 0x7f16cc101f90 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.995+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f16c4011ac0 con 0x7f16cc101f90 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.995+0000 7f16c9ffb640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 0x7f16b80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.995+0000 7f16cbfff640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 0x7f16b80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.995+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f16c40cdd60 con 0x7f16cc101f90 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.996+0000 7f16cbfff640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 0x7f16b80a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f16bc005ab0 tx=0x7f16bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:53.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:52.999+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f16c4096d50 con 0x7f16cc101f90 2024-03-16T20:33:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.181+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f16cc10cf20 con 0x7f16cc101f90 2024-03-16T20:33:53.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:52 smithi031 ceph-mon[36281]: pgmap v288: 97 pgs: 13 creating+peering, 15 unknown, 69 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:53.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:52 smithi031 ceph-mon[36281]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T20:33:53.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:52 smithi189 ceph-mon[36655]: pgmap v288: 97 pgs: 13 creating+peering, 15 unknown, 69 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:53.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:52 smithi189 ceph-mon[36655]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T20:33:53.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:52 smithi023 ceph-mon[31879]: pgmap v288: 97 pgs: 13 creating+peering, 15 unknown, 69 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:53.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:52 smithi023 ceph-mon[31879]: osdmap e87: 12 total, 12 up, 12 in 2024-03-16T20:33:53.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.866+0000 7f16c9ffb640 1 -- 172.21.15.23:0/1217390323 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7f16c409a410 con 0x7f16cc101f90 2024-03-16T20:33:53.871 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2024-03-16T20:33:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 msgr2=0x7f16b80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 0x7f16b80a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f16bc005ab0 tx=0x7f16bc005fd0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 msgr2=0x7f16cc101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f16c400eae0 tx=0x7f16c400c270 comp rx=0 tx=0).stop 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 shutdown_connections 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f16b80a1a30 0x7f16b80a3ef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc1071f0 0x7f16cc100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16cc1048c0 0x7f16cc0fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 --2- 172.21.15.23:0/1217390323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16cc101f90 0x7f16cc101900 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 >> 172.21.15.23:0/1217390323 conn(0x7f16cc0fb9a0 msgr2=0x7f16cc102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 shutdown_connections 2024-03-16T20:33:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:53.870+0000 7f16d2b3a640 1 -- 172.21.15.23:0/1217390323 wait complete. 2024-03-16T20:33:53.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-16T20:33:54.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 -- 172.21.15.23:0/2338481672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5984102700 msgr2=0x7f5984102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:54.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 --2- 172.21.15.23:0/2338481672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5984102700 0x7f5984102b80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5978009900 tx=0x7f597802bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 -- 172.21.15.23:0/2338481672 shutdown_connections 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 --2- 172.21.15.23:0/2338481672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59841030c0 0x7f5984109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 --2- 172.21.15.23:0/2338481672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5984102700 0x7f5984102b80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 --2- 172.21.15.23:0/2338481672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 0x7f5984101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 -- 172.21.15.23:0/2338481672 >> 172.21.15.23:0/2338481672 conn(0x7f59840fcaa0 msgr2=0x7f59840feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 -- 172.21.15.23:0/2338481672 shutdown_connections 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.005+0000 7f598b284640 1 -- 172.21.15.23:0/2338481672 wait complete. 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 Processor -- start 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 -- start start 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 0x7f59840752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5984102700 0x7f59840738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984074580 con 0x7f59841030c0 2024-03-16T20:33:54.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840746f0 con 0x7f5984102700 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59841a15d0 con 0x7f5984101510 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46310/0 (socket says 172.21.15.23:46310) 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 -- 172.21.15.23:0/1996091982 learned_addr learned my addr 172.21.15.23:0/1996091982 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989020640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 0x7f59840752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598881f640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5984102700 0x7f59840738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 -- 172.21.15.23:0/1996091982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 msgr2=0x7f59840752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 0x7f59840752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 -- 172.21.15.23:0/1996091982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5984102700 msgr2=0x7f59840738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5984102700 0x7f59840738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 -- 172.21.15.23:0/1996091982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978009560 con 0x7f59841030c0 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f5989821640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f597c00eb40 tx=0x7f597c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:54.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c004b20 con 0x7f59841030c0 2024-03-16T20:33:54.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f597c004c80 con 0x7f59841030c0 2024-03-16T20:33:54.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.006+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59841a1880 con 0x7f59841030c0 2024-03-16T20:33:54.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.007+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c010670 con 0x7f59841030c0 2024-03-16T20:33:54.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.007+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841a1d10 con 0x7f59841030c0 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.008+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984108830 con 0x7f59841030c0 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.009+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f597c005020 con 0x7f59841030c0 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.009+0000 7f59767fc640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 0x7f59700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.009+0000 7f5989020640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 0x7f59700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.009+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f597c014070 con 0x7f59841030c0 2024-03-16T20:33:54.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.010+0000 7f5989020640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 0x7f59700a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5984102560 tx=0x7f5980005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:54.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.014+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f597c096bd0 con 0x7f59841030c0 2024-03-16T20:33:54.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:53 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T20:33:54.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:53 smithi031 ceph-mon[36281]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T20:33:54.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:53 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T20:33:54.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:53 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T20:33:54.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:53 smithi189 ceph-mon[36655]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T20:33:54.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:53 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T20:33:54.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.238+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f598402d000 con 0x7f59841030c0 2024-03-16T20:33:54.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:53 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4276440795' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-16T20:33:54.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:53 smithi023 ceph-mon[31879]: osdmap e88: 12 total, 12 up, 12 in 2024-03-16T20:33:54.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:53 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-16T20:33:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.922+0000 7f59767fc640 1 -- 172.21.15.23:0/1996091982 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f597c09a290 con 0x7f59841030c0 2024-03-16T20:33:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 4 to fsmap 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 msgr2=0x7f59700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 0x7f59700a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5984102560 tx=0x7f5980005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 msgr2=0x7f5984073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f597c00eb40 tx=0x7f597c00c2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 shutdown_connections 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f59700a1a30 0x7f59700a3ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59841030c0 0x7f5984073e20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5984102700 0x7f59840738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 --2- 172.21.15.23:0/1996091982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5984101510 0x7f59840752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.925+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 >> 172.21.15.23:0/1996091982 conn(0x7f59840fcaa0 msgr2=0x7f5984105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.926+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 shutdown_connections 2024-03-16T20:33:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:54.926+0000 7f598b284640 1 -- 172.21.15.23:0/1996091982 wait complete. 2024-03-16T20:33:54.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/3806233594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34102040 msgr2=0x7f3b341088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/3806233594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34102040 0x7f3b341088d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009ca0 tx=0x7f3b2c02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/3806233594 shutdown_connections 2024-03-16T20:33:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/3806233594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34102040 0x7f3b341088d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/3806233594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34101680 0x7f3b34101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/3806233594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b34100490 0x7f3b341008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/3806233594 >> 172.21.15.23:0/3806233594 conn(0x7f3b340fba20 msgr2=0x7f3b340fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/3806233594 shutdown_connections 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.047+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/3806233594 wait complete. 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 Processor -- start 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 -- start start 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34100490 0x7f3b34197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b34102040 0x7f3b3419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419f010 con 0x7f3b34101680 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419f180 con 0x7f3b34102040 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b3a7d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419f340 con 0x7f3b34100490 2024-03-16T20:33:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46332/0 (socket says 172.21.15.23:46332) 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b337fe640 1 -- 172.21.15.23:0/1835437168 learned_addr learned my addr 172.21.15.23:0/1835437168 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.048+0000 7f3b33fff640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34100490 0x7f3b34197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 -- 172.21.15.23:0/1835437168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34100490 msgr2=0x7f3b34197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34100490 0x7f3b34197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 -- 172.21.15.23:0/1835437168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b34102040 msgr2=0x7f3b3419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b34102040 0x7f3b3419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 -- 172.21.15.23:0/1835437168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009900 con 0x7f3b34101680 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b337fe640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800e9f0 tx=0x7f3b2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28009720 con 0x7f3b34101680 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b2800cd80 con 0x7f3b34101680 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3419f620 con 0x7f3b34101680 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28010690 con 0x7f3b34101680 2024-03-16T20:33:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.049+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3419fae0 con 0x7f3b34101680 2024-03-16T20:33:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.050+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3410be30 con 0x7f3b34101680 2024-03-16T20:33:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.051+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3b280046c0 con 0x7f3b34101680 2024-03-16T20:33:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.051+0000 7f3b317fa640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 0x7f3b1c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.052+0000 7f3b33fff640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 0x7f3b1c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.052+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3b28014070 con 0x7f3b34101680 2024-03-16T20:33:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.052+0000 7f3b33fff640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 0x7f3b1c0a3ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b341014e0 tx=0x7f3b2400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.056+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3b28096a30 con 0x7f3b34101680 2024-03-16T20:33:55.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:54 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T20:33:55.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:54 smithi031 ceph-mon[36281]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T20:33:55.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:54 smithi031 ceph-mon[36281]: pgmap v291: 129 pgs: 16 creating+peering, 29 unknown, 84 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:55.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:54 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T20:33:55.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:54 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T20:33:55.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:54 smithi189 ceph-mon[36655]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T20:33:55.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:54 smithi189 ceph-mon[36655]: pgmap v291: 129 pgs: 16 creating+peering, 29 unknown, 84 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:55.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:54 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T20:33:55.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.279+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b3419fd60 con 0x7f3b34101680 2024-03-16T20:33:55.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.280+0000 7f3b317fa640 1 -- 172.21.15.23:0/1835437168 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11315 (secure 0 0 0) 0x7f3b2809a0f0 con 0x7f3b34101680 2024-03-16T20:33:55.285 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24497,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:33:54.239386+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24511,"qdb_cluster":[24511]},"id":1}]} 2024-03-16T20:33:55.286 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2024-03-16T20:33:55.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 msgr2=0x7f3b1c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 0x7f3b1c0a3ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b341014e0 tx=0x7f3b2400b390 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 msgr2=0x7f3b34197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800e9f0 tx=0x7f3b2800edf0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 shutdown_connections 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b1c0a1a30 0x7f3b1c0a3ef0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b34102040 0x7f3b3419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34101680 0x7f3b34197a60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 --2- 172.21.15.23:0/1835437168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b34100490 0x7f3b34197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 >> 172.21.15.23:0/1835437168 conn(0x7f3b340fba20 msgr2=0x7f3b34069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 shutdown_connections 2024-03-16T20:33:55.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.282+0000 7f3b3a7d4640 1 -- 172.21.15.23:0/1835437168 wait complete. 2024-03-16T20:33:55.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:54 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1217390323' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-16T20:33:55.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:54 smithi023 ceph-mon[31879]: osdmap e89: 12 total, 12 up, 12 in 2024-03-16T20:33:55.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:54 smithi023 ceph-mon[31879]: pgmap v291: 129 pgs: 16 creating+peering, 29 unknown, 84 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-16T20:33:55.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:54 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-16T20:33:55.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:33:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.395+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/3426250861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101fe0 msgr2=0x7f9f98108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.395+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/3426250861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101fe0 0x7f9f98108870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009900 tx=0x7f9f9002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/3426250861 shutdown_connections 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/3426250861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101fe0 0x7f9f98108870 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/3426250861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98101620 0x7f9f98101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/3426250861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98100430 0x7f9f98100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/3426250861 >> 172.21.15.23:0/3426250861 conn(0x7f9f980fb9a0 msgr2=0x7f9f980fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/3426250861 shutdown_connections 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/3426250861 wait complete. 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 Processor -- start 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- start start 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98100430 0x7f9f98197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98101fe0 0x7f9f9819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.396+0000 7f9f9dd5c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9819f030 con 0x7f9f98101620 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f9dd5c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9819f1a0 con 0x7f9f98101fe0 2024-03-16T20:33:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f9dd5c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9819f360 con 0x7f9f98100430 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46352/0 (socket says 172.21.15.23:46352) 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 -- 172.21.15.23:0/2509206863 learned_addr learned my addr 172.21.15.23:0/2509206863 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f977fe640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98100430 0x7f9f98197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f97fff640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98101fe0 0x7f9f9819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 -- 172.21.15.23:0/2509206863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98100430 msgr2=0x7f9f98197540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98100430 0x7f9f98197540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 -- 172.21.15.23:0/2509206863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98101fe0 msgr2=0x7f9f9819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98101fe0 0x7f9f9819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 -- 172.21.15.23:0/2509206863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f90009560 con 0x7f9f98101620 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f96ffd640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00e9f0 tx=0x7f9f8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c009700 con 0x7f9f98101620 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c00cd80 con 0x7f9f98101620 2024-03-16T20:33:55.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9819f640 con 0x7f9f98101620 2024-03-16T20:33:55.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c018920 con 0x7f9f98101620 2024-03-16T20:33:55.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.397+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9819fb50 con 0x7f9f98101620 2024-03-16T20:33:55.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.398+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9810be40 con 0x7f9f98101620 2024-03-16T20:33:55.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.399+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9f8c018a80 con 0x7f9f98101620 2024-03-16T20:33:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.400+0000 7f9f94ff9640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 0x7f9f800a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.400+0000 7f9f977fe640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 0x7f9f800a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.400+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9f8c014070 con 0x7f9f98101620 2024-03-16T20:33:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.400+0000 7f9f977fe640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 0x7f9f800a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9f98101480 tx=0x7f9f8800a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.404+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9f8c053280 con 0x7f9f98101620 2024-03-16T20:33:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.589+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f9810bfd0 con 0x7f9f98101620 2024-03-16T20:33:55.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.591+0000 7f9f94ff9640 1 -- 172.21.15.23:0/2509206863 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f9f8c096a30 con 0x7f9f98101620 2024-03-16T20:33:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:55.597 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:55.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 msgr2=0x7f9f800a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 0x7f9f800a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9f98101480 tx=0x7f9f8800a590 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 msgr2=0x7f9f98197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00e9f0 tx=0x7f9f8c00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 shutdown_connections 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f800a1a30 0x7f9f800a3ef0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f98101fe0 0x7f9f9819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f98101620 0x7f9f98197a80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 --2- 172.21.15.23:0/2509206863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f98100430 0x7f9f98197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 >> 172.21.15.23:0/2509206863 conn(0x7f9f980fb9a0 msgr2=0x7f9f980fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 shutdown_connections 2024-03-16T20:33:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.593+0000 7f9f9dd5c640 1 -- 172.21.15.23:0/2509206863 wait complete. 2024-03-16T20:33:55.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:33:55.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 -- 172.21.15.23:0/714199294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 msgr2=0x7f6878108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 --2- 172.21.15.23:0/714199294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f6878108850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6870009ca0 tx=0x7f687002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 -- 172.21.15.23:0/714199294 shutdown_connections 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 --2- 172.21.15.23:0/714199294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f6878108850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 --2- 172.21.15.23:0/714199294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 0x7f6878101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 --2- 172.21.15.23:0/714199294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.706+0000 7f687d73a640 1 -- 172.21.15.23:0/714199294 >> 172.21.15.23:0/714199294 conn(0x7f68780fb9a0 msgr2=0x7f68780fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- 172.21.15.23:0/714199294 shutdown_connections 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- 172.21.15.23:0/714199294 wait complete. 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 Processor -- start 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- start start 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 0x7f6878197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687819f0e0 con 0x7f6878101fc0 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687819f250 con 0x7f6878101600 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f687d73a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687819f410 con 0x7f6878100410 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f68777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f68777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46366/0 (socket says 172.21.15.23:46366) 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f68777fe640 1 -- 172.21.15.23:0/307357785 learned_addr learned my addr 172.21.15.23:0/307357785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.707+0000 7f6876ffd640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 -- 172.21.15.23:0/307357785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 msgr2=0x7f6878197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f686ffff640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 0x7f6878197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 -- 172.21.15.23:0/307357785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 msgr2=0x7f6878197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 0x7f6878197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 -- 172.21.15.23:0/307357785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870009900 con 0x7f6878101fc0 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f6876ffd640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:33:55.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f68777fe640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f687002c630 tx=0x7f6870034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870032eb0 con 0x7f6878101fc0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6870003e50 con 0x7f6878101fc0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687819f690 con 0x7f6878101fc0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870040450 con 0x7f6878101fc0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.708+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687819fb40 con 0x7f6878101fc0 2024-03-16T20:33:55.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.709+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687810be80 con 0x7f6878101fc0 2024-03-16T20:33:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.710+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f687000a040 con 0x7f6878101fc0 2024-03-16T20:33:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.711+0000 7f6874ff9640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 0x7f68640a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.711+0000 7f6876ffd640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 0x7f68640a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.711+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f68700f0a30 con 0x7f6878101fc0 2024-03-16T20:33:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.711+0000 7f6876ffd640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 0x7f68640a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6878101460 tx=0x7f6868005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:55.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.714+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f68700b9a00 con 0x7f6878101fc0 2024-03-16T20:33:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.900+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f687819ffe0 con 0x7f6878101fc0 2024-03-16T20:33:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.901+0000 7f6874ff9640 1 -- 172.21.15.23:0/307357785 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f68700bd0c0 con 0x7f6878101fc0 2024-03-16T20:33:55.907 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:55.907 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 msgr2=0x7f68640a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 0x7f68640a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6878101460 tx=0x7f6868005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 msgr2=0x7f687819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f687002c630 tx=0x7f6870034cc0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 shutdown_connections 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68640a1a30 0x7f68640a3ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6878101fc0 0x7f687819eba0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6878101600 0x7f6878197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 --2- 172.21.15.23:0/307357785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6878100410 0x7f6878197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 >> 172.21.15.23:0/307357785 conn(0x7f68780fb9a0 msgr2=0x7f68780fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 shutdown_connections 2024-03-16T20:33:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:55.903+0000 7f687d73a640 1 -- 172.21.15.23:0/307357785 wait complete. 2024-03-16T20:33:55.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:33:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 -- 172.21.15.23:0/1413484231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 msgr2=0x7f9234100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 --2- 172.21.15.23:0/1413484231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234100260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9230009900 tx=0x7f923002bc80 comp rx=0 tx=0).stop 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 -- 172.21.15.23:0/1413484231 shutdown_connections 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 --2- 172.21.15.23:0/1413484231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9234101140 0x7f92341015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 --2- 172.21.15.23:0/1413484231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92341007a0 0x7f9234100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 --2- 172.21.15.23:0/1413484231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234100260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 -- 172.21.15.23:0/1413484231 >> 172.21.15.23:0/1413484231 conn(0x7f92340fba20 msgr2=0x7f92340fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 -- 172.21.15.23:0/1413484231 shutdown_connections 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.030+0000 7f923b888640 1 -- 172.21.15.23:0/1413484231 wait complete. 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 Processor -- start 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 -- start start 2024-03-16T20:33:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 0x7f9234199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9234101140 0x7f92341a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341a1280 con 0x7f92340ffe40 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341a13f0 con 0x7f92341007a0 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341a15b0 con 0x7f9234101140 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46378/0 (socket says 172.21.15.23:46378) 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 -- 172.21.15.23:0/2585311037 learned_addr learned my addr 172.21.15.23:0/2585311037 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239e25640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9234101140 0x7f92341a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9238e23640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 0x7f9234199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 -- 172.21.15.23:0/2585311037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9234101140 msgr2=0x7f92341a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9234101140 0x7f92341a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 -- 172.21.15.23:0/2585311037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 msgr2=0x7f9234199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 0x7f9234199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 -- 172.21.15.23:0/2585311037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9224009490 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9238e23640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 0x7f9234199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f9239624640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f923002c690 tx=0x7f92300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923002dea0 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92300044e0 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9230009560 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.031+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230003d70 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.032+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92341a1c00 con 0x7f92340ffe40 2024-03-16T20:33:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.033+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9234068730 con 0x7f92340ffe40 2024-03-16T20:33:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.033+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9230005000 con 0x7f92340ffe40 2024-03-16T20:33:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.034+0000 7f922a7fc640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 0x7f92200a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.034+0000 7f9238e23640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 0x7f92200a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.034+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f92300efde0 con 0x7f92340ffe40 2024-03-16T20:33:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.034+0000 7f9238e23640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 0x7f92200a3f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f923419aca0 tx=0x7f9224005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.038+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f92300b8db0 con 0x7f92340ffe40 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1835437168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:55 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2509206863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:33:56 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:33:56.037+0000 7f8743f82700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1835437168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:55 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2509206863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.260+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f923410f270 con 0x7f92340ffe40 2024-03-16T20:33:56.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.261+0000 7f922a7fc640 1 -- 172.21.15.23:0/2585311037 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11315 (secure 0 0 0) 0x7f9230043020 con 0x7f92340ffe40 2024-03-16T20:33:56.266 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:56.266 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24493,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24497,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:33:54.239386+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24511,"qdb_cluster":[24511]},"id":1}]} 2024-03-16T20:33:56.269 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 msgr2=0x7f92200a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 0x7f92200a3f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f923419aca0 tx=0x7f9224005f90 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 msgr2=0x7f9234199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f923002c690 tx=0x7f92300042f0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 shutdown_connections 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92200a1ab0 0x7f92200a3f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9234101140 0x7f92341a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92341007a0 0x7f9234199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 --2- 172.21.15.23:0/2585311037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92340ffe40 0x7f9234199790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 >> 172.21.15.23:0/2585311037 conn(0x7f92340fba20 msgr2=0x7f92340fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 shutdown_connections 2024-03-16T20:33:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.263+0000 7f923b888640 1 -- 172.21.15.23:0/2585311037 wait complete. 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: osdmap e90: 12 total, 12 up, 12 in 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1996091982' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1835437168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2509206863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:56.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 -- 172.21.15.23:0/2983261563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e40ffe00 msgr2=0x7f08e4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 --2- 172.21.15.23:0/2983261563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e40ffe00 0x7f08e4102230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009900 tx=0x7f08e002bc30 comp rx=0 tx=0).stop 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 -- 172.21.15.23:0/2983261563 shutdown_connections 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 --2- 172.21.15.23:0/2983261563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e4105120 0x7f08e41055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 --2- 172.21.15.23:0/2983261563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 0x7f08e4104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 --2- 172.21.15.23:0/2983261563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e40ffe00 0x7f08e4102230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 -- 172.21.15.23:0/2983261563 >> 172.21.15.23:0/2983261563 conn(0x7f08e40fba20 msgr2=0x7f08e40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 -- 172.21.15.23:0/2983261563 shutdown_connections 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.376+0000 7f08ec047640 1 -- 172.21.15.23:0/2983261563 wait complete. 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 Processor -- start 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 -- start start 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e40ffe00 0x7f08e419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 0x7f08e419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41a5690 con 0x7f08e4105120 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41a5800 con 0x7f08e40ffe00 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ec047640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41a59c0 con 0x7f08e4102770 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46400/0 (socket says 172.21.15.23:46400) 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 -- 172.21.15.23:0/2509535626 learned_addr learned my addr 172.21.15.23:0/2509535626 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08e95e2640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 0x7f08e419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08e9de3640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e40ffe00 0x7f08e419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 -- 172.21.15.23:0/2509535626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 msgr2=0x7f08e419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 0x7f08e419e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 -- 172.21.15.23:0/2509535626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e40ffe00 msgr2=0x7f08e419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.377+0000 7f08ea5e4640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e40ffe00 0x7f08e419dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08ea5e4640 1 -- 172.21.15.23:0/2509535626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0009560 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08ea5e4640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00eb10 tx=0x7f08dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc004b20 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e41a5ca0 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08dc004c80 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc010650 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.378+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e41a6160 con 0x7f08e4105120 2024-03-16T20:33:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.379+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e40fd510 con 0x7f08e4105120 2024-03-16T20:33:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.380+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f08dc0107f0 con 0x7f08e4105120 2024-03-16T20:33:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.380+0000 7f08daffd640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 0x7f08d00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.380+0000 7f08e9de3640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 0x7f08d00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.381+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f08dc014070 con 0x7f08e4105120 2024-03-16T20:33:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.381+0000 7f08e9de3640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 0x7f08d00a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08e002c840 tx=0x7f08e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.384+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f08dc096de0 con 0x7f08e4105120 2024-03-16T20:33:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.568+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08e41136f0 con 0x7f08e4105120 2024-03-16T20:33:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.570+0000 7f08daffd640 1 -- 172.21.15.23:0/2509535626 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f08dc09a4a0 con 0x7f08e4105120 2024-03-16T20:33:56.576 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:33:56.576 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:33:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 msgr2=0x7f08d00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 0x7f08d00a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08e002c840 tx=0x7f08e0000bc0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 msgr2=0x7f08e41a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00eb10 tx=0x7f08dc00c2d0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 shutdown_connections 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08d00a1a30 0x7f08d00a3ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e4105120 0x7f08e41a5150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e4102770 0x7f08e419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 --2- 172.21.15.23:0/2509535626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08e40ffe00 0x7f08e419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 >> 172.21.15.23:0/2509535626 conn(0x7f08e40fba20 msgr2=0x7f08e41030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 shutdown_connections 2024-03-16T20:33:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.572+0000 7f08ec047640 1 -- 172.21.15.23:0/2509535626 wait complete. 2024-03-16T20:33:56.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-16T20:33:49.239658+0000', 'modified': '2024-03-16T20:33:54.239386+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24511}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24511': {'gid': 24511, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.189:6839/2333060283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2333060283}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2333060283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24511, 'qdb_cluster': [24511]} 2024-03-16T20:33:56.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-16T20:33:56.589 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T20:33:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 -- 172.21.15.23:0/2500885374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118103160 msgr2=0x7f91181099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 --2- 172.21.15.23:0/2500885374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118103160 0x7f91181099f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9110009ca0 tx=0x7f911002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 -- 172.21.15.23:0/2500885374 shutdown_connections 2024-03-16T20:33:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 --2- 172.21.15.23:0/2500885374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118103160 0x7f91181099f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 --2- 172.21.15.23:0/2500885374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91181027a0 0x7f9118102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 --2- 172.21.15.23:0/2500885374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91180ff7d0 0x7f91180ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 -- 172.21.15.23:0/2500885374 >> 172.21.15.23:0/2500885374 conn(0x7f91180fb0e0 msgr2=0x7f91180fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 -- 172.21.15.23:0/2500885374 shutdown_connections 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.687+0000 7f911f8f4640 1 -- 172.21.15.23:0/2500885374 wait complete. 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 Processor -- start 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 -- start start 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91181027a0 0x7f9118100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 0x7f9118100c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181013f0 con 0x7f91180ff7d0 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118101560 con 0x7f9118103160 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911f8f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181a1580 con 0x7f91181027a0 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46420/0 (socket says 172.21.15.23:46420) 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 -- 172.21.15.23:0/463086959 learned_addr learned my addr 172.21.15.23:0/463086959 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911ce8f640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91181027a0 0x7f9118100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911de91640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 0x7f9118100c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 -- 172.21.15.23:0/463086959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91181027a0 msgr2=0x7f9118100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91181027a0 0x7f9118100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 -- 172.21.15.23:0/463086959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 msgr2=0x7f9118100c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 0x7f9118100c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 -- 172.21.15.23:0/463086959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110009900 con 0x7f91180ff7d0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911de91640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 0x7f9118100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.688+0000 7f911d690640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f911400e9f0 tx=0x7f911400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.689+0000 7f910e7fc640 1 -- 172.21.15.23:0/463086959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91140096e0 con 0x7f91180ff7d0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.689+0000 7f910e7fc640 1 -- 172.21.15.23:0/463086959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f911400cd80 con 0x7f91180ff7d0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.689+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91181a1830 con 0x7f91180ff7d0 2024-03-16T20:33:56.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.689+0000 7f910e7fc640 1 -- 172.21.15.23:0/463086959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114010430 con 0x7f91180ff7d0 2024-03-16T20:33:56.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.689+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91181a1cf0 con 0x7f91180ff7d0 2024-03-16T20:33:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f910e7fc640 1 -- 172.21.15.23:0/463086959 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9114003680 con 0x7f91180ff7d0 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 Processor -- start 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 -- start start 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe65470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe67f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fe1bcd0 con 0x555a4fe5ff80 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fdcef20 con 0x555a4fe60970 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f91225af740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fdcc450 con 0x555a4fe65ae0 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f911de91640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f911de91640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46424/0 (socket says 172.21.15.23:46424) 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f911de91640 1 -- 172.21.15.23:0/3042070206 learned_addr learned my addr 172.21.15.23:0/3042070206 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f911ce8f640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe67f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.691+0000 7f911d690640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe65470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f910e7fc640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 0x7f91040a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 -- 172.21.15.23:0/3042070206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 msgr2=0x555a4fe67f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe67f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 -- 172.21.15.23:0/3042070206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 msgr2=0x555a4fe65470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe65470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 -- 172.21.15.23:0/3042070206 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a4fd7cf20 con 0x555a4fe5ff80 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911d690640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe65470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 0x7f91040a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f910e7fc640 1 -- 172.21.15.23:0/463086959 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9114014070 con 0x7f91180ff7d0 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f91100323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c99203e5ff9e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f91037fe640 1 -- 172.21.15.23:0/3042070206 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110034640 con 0x555a4fe5ff80 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f91037fe640 1 -- 172.21.15.23:0/3042070206 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91100347a0 con 0x555a4fe5ff80 2024-03-16T20:33:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f911de91640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 0x7f91040a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9110006010 tx=0x7f911004c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.692+0000 7f91037fe640 1 -- 172.21.15.23:0/3042070206 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91100042a0 con 0x555a4fe5ff80 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- 172.21.15.23:0/3042070206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 msgr2=0x555a4fe603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f91100323c0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- 172.21.15.23:0/3042070206 shutdown_connections 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe67f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe65470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 --2- 172.21.15.23:0/3042070206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe603a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- 172.21.15.23:0/3042070206 >> 172.21.15.23:0/3042070206 conn(0x555a4fe5b4f0 msgr2=0x555a4fe5b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- 172.21.15.23:0/3042070206 shutdown_connections 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- 172.21.15.23:0/3042070206 wait complete. 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 -- start start 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.693+0000 7f91225af740 1 Processor -- start 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe60970 0x555a4fe6a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe65ae0 0x555a4fe716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fdc7c30 con 0x555a4fe5ff80 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fd6cd30 con 0x555a4fe65ae0 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fa9b310 con 0x555a4fe60970 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 10 client.0 _ll_register_callbacks cb 0x555a4fd74d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46438/0 (socket says 172.21.15.23:46438) 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 -- 172.21.15.23:0/1311917459 learned_addr learned my addr 172.21.15.23:0/1311917459 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911de91640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe65ae0 0x555a4fe716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911ce8f640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe60970 0x555a4fe6a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 -- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe60970 msgr2=0x555a4fe6a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe60970 0x555a4fe6a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 -- 172.21.15.23:0/1311917459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe65ae0 msgr2=0x555a4fe716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe65ae0 0x555a4fe716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 -- 172.21.15.23:0/1311917459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911810a230 con 0x555a4fe5ff80 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f911d690640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9114004630 tx=0x7f911409a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2595c1817f4fddc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f9100ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911409a7b0 con 0x555a4fe5ff80 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f911409e900 con 0x555a4fe5ff80 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.694+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a4fb18270 con 0x555a4fe5ff80 2024-03-16T20:33:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91140a2f50 con 0x555a4fe5ff80 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555a4fdc62b0 con 0x555a4fe5ff80 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555a4fb18270 con 0x555a4fe5ff80 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f911401f060 con 0x555a4fe5ff80 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 10 client.14775 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 20 client.14775 populate_metadata read hostname 'smithi023' 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fe76970 0x555a4fe7adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 4 client.14775 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a4fb04610 con 0x555a4fe76970 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f911ce8f640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fe76970 0x555a4fe7adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f9100ff9640 10 client.14775 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.695+0000 7f911ce8f640 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fe76970 0x555a4fe7adf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7816bb32585eb786 server_cookie=d404ab95ad2a83ae in_seq=0 out_seq=0 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f9100ff9640 1 -- 172.21.15.23:0/1311917459 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a4fb04610 con 0x555a4fe76970 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f9100ff9640 10 client.14775 handle_command_reply: tid=0 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 client.14775 shutdown 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 msgr2=0x555a4fe6a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9114004630 tx=0x7f911409a3b0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 shutdown_connections 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fe76970 0x555a4fe7adf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe65ae0 0x555a4fe716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f9100ff9640 0 client.14775 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe60970 0x555a4fe6a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 --2- 172.21.15.23:0/1311917459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 >> 172.21.15.23:0/1311917459 conn(0x555a4fe5b4f0 msgr2=0x555a4fe5a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 shutdown_connections 2024-03-16T20:33:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 1 -- 172.21.15.23:0/1311917459 wait complete. 2024-03-16T20:33:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.696+0000 7f91225af740 20 client.14775 trim_cache size 0 max 16384 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 Processor -- start 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 -- start start 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe71680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe5b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fd4d490 con 0x555a4fe5ff80 2024-03-16T20:33:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fb04610 con 0x555a4fe60970 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f91225af740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fb18270 con 0x555a4fe65ae0 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46442/0 (socket says 172.21.15.23:46442) 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 -- 172.21.15.23:0/2183025382 learned_addr learned my addr 172.21.15.23:0/2183025382 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911de91640 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe5b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 -- 172.21.15.23:0/2183025382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 msgr2=0x555a4fe5b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe5b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 -- 172.21.15.23:0/2183025382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 msgr2=0x555a4fe71680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe71680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.713+0000 7f911d690640 1 -- 172.21.15.23:0/2183025382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a4fd62a20 con 0x555a4fe5ff80 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f911d690640 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9114098a60 tx=0x7f911409a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a3f4d585b06e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f9102ffd640 1 -- 172.21.15.23:0/2183025382 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91140ba0b0 con 0x555a4fe5ff80 2024-03-16T20:33:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f9102ffd640 1 -- 172.21.15.23:0/2183025382 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9114096a50 con 0x555a4fe5ff80 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f9102ffd640 1 -- 172.21.15.23:0/2183025382 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91140a2f50 con 0x555a4fe5ff80 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 -- 172.21.15.23:0/2183025382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 msgr2=0x555a4fe6a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9114098a60 tx=0x7f911409a420 comp rx=0 tx=0).stop 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 -- 172.21.15.23:0/2183025382 shutdown_connections 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe5b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe60970 0x555a4fe71680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 --2- 172.21.15.23:0/2183025382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe5ff80 0x555a4fe6a610 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 -- 172.21.15.23:0/2183025382 >> 172.21.15.23:0/2183025382 conn(0x555a4fe5b4f0 msgr2=0x555a4fe56660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 -- 172.21.15.23:0/2183025382 shutdown_connections 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.714+0000 7f91225af740 1 -- 172.21.15.23:0/2183025382 wait complete. 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 -- start start 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 Processor -- start 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 0x555a4fe713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe6e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fe6e8e0 con 0x555a4fe60970 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fe6ea50 con 0x555a4fe5ff80 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a4fed1f10 con 0x555a4fe65ae0 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f91225af740 10 client.0 _ll_register_callbacks cb 0x7f9114003980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46454/0 (socket says 172.21.15.23:46454) 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 -- 172.21.15.23:0/3524263399 learned_addr learned my addr 172.21.15.23:0/3524263399 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911de91640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe6e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911d690640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 0x555a4fe713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 -- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 msgr2=0x555a4fe6e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe6e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 -- 172.21.15.23:0/3524263399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 msgr2=0x555a4fe713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:33:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 0x555a4fe713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911ce8f640 1 -- 172.21.15.23:0/3524263399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a4fdc62b0 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.715+0000 7f911d690640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 0x555a4fe713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f911ce8f640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x555a4fe5c460 tx=0x7f910800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b6caa95a8e1a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108016070 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f91080035f0 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a4fe779a0 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91080037b0 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 10 client.14781 fetch_fsmap learned FSMap version 17 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555a4fd88780 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555a4fe779a0 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f9108011a40 con 0x555a4fe60970 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 10 client.14781 fetch_fsmap finished waiting for FSMap version 17 2024-03-16T20:33:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 10 client.14781 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 20 client.14781 populate_metadata read hostname 'smithi023' 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fed7270 0x555a4fed36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 4 client.14781 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555a4fac76c0 con 0x555a4fed7270 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.716+0000 7f911d690640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fed7270 0x555a4fed36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f9100ff9640 10 client.14781 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f911d690640 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fed7270 0x555a4fed36f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f1581477d9b16d server_cookie=96e681a212f9c528 in_seq=0 out_seq=0 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f9100ff9640 1 -- 172.21.15.23:0/3524263399 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x555a4fac76c0 con 0x555a4fed7270 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f9100ff9640 10 client.14781 handle_command_reply: tid=0 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 client.14781 shutdown 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 msgr2=0x555a4fe6de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x555a4fe5c460 tx=0x7f910800be30 comp rx=0 tx=0).stop 2024-03-16T20:33:56.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 shutdown_connections 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x555a4fed7270 0x555a4fed36f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a4fe65ae0 0x555a4fe6e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f9100ff9640 0 client.14781 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555a4fe60970 0x555a4fe6de60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 --2- 172.21.15.23:0/3524263399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a4fe5ff80 0x555a4fe713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 >> 172.21.15.23:0/3524263399 conn(0x555a4fe5b4f0 msgr2=0x555a4fe5b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 shutdown_connections 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 1 -- 172.21.15.23:0/3524263399 wait complete. 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.717+0000 7f91225af740 20 client.14781 trim_cache size 0 max 16384 2024-03-16T20:33:56.731 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24511, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 6.5074902669999997, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 17, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 20.752704238 2024-03-16T20:33:56.732 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 msgr2=0x7f91040a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 0x7f91040a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9110006010 tx=0x7f911004c000 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 msgr2=0x7f9118102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f911400e9f0 tx=0x7f911400eeb0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 shutdown_connections 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f91040a1a30 0x7f91040a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9118103160 0x7f9118100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91181027a0 0x7f9118100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 --2- 172.21.15.23:0/463086959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91180ff7d0 0x7f9118102110 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 >> 172.21.15.23:0/463086959 conn(0x7f91180fb0e0 msgr2=0x7f9118105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 shutdown_connections 2024-03-16T20:33:56.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:56.718+0000 7f911f8f4640 1 -- 172.21.15.23:0/463086959 wait complete. 2024-03-16T20:33:56.735 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T20:33:56.735 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:33:56.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T20:33:56.978 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:33:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/307357785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:56 smithi031 ceph-mon[36281]: pgmap v293: 129 pgs: 3 creating+peering, 9 unknown, 117 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-16T20:33:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2585311037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2509535626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/307357785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:56 smithi189 ceph-mon[36655]: pgmap v293: 129 pgs: 3 creating+peering, 9 unknown, 117 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-16T20:33:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2585311037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2509535626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/307357785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:56 smithi023 ceph-mon[31879]: pgmap v293: 129 pgs: 3 creating+peering, 9 unknown, 117 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-16T20:33:57.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2585311037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:33:57.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2509535626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:33:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:57 smithi031 ceph-mon[36281]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:58.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:57 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:33:58.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:57 smithi189 ceph-mon[36655]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:58.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:57 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:33:58.254 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:57 smithi023 ceph-mon[31879]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T20:33:58.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:57 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:33:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.607+0000 7f2cc4573700 1 -- 172.21.15.23:0/1159942209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc107300 msgr2=0x7f2cbc10eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.607+0000 7f2cc4573700 1 --2- 172.21.15.23:0/1159942209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc107300 0x7f2cbc10eed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400b0a0 tx=0x7f2cb400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 -- 172.21.15.23:0/1159942209 shutdown_connections 2024-03-16T20:33:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 --2- 172.21.15.23:0/1159942209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc107300 0x7f2cbc10eed0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 --2- 172.21.15.23:0/1159942209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc102220 0x7f2cbc102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 --2- 172.21.15.23:0/1159942209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 -- 172.21.15.23:0/1159942209 >> 172.21.15.23:0/1159942209 conn(0x7f2cbc0fda90 msgr2=0x7f2cbc0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 -- 172.21.15.23:0/1159942209 shutdown_connections 2024-03-16T20:33:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 -- 172.21.15.23:0/1159942209 wait complete. 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 Processor -- start 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.608+0000 7f2cc4573700 1 -- start start 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102220 0x7f2cbc198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc107300 0x7f2cbc19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d4e0 con 0x7f2cbc102bd0 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d650 con 0x7f2cbc107300 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19d810 con 0x7f2cbc102220 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46476/0 (socket says 172.21.15.23:46476) 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 -- 172.21.15.23:0/3824747760 learned_addr learned my addr 172.21.15.23:0/3824747760 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:33:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc230f700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102220 0x7f2cbc198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc2b10700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc107300 0x7f2cbc19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 -- 172.21.15.23:0/3824747760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102220 msgr2=0x7f2cbc198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102220 0x7f2cbc198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 -- 172.21.15.23:0/3824747760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc107300 msgr2=0x7f2cbc19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc107300 0x7f2cbc19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 -- 172.21.15.23:0/3824747760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8009520 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc1b0e700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800ea30 tx=0x7f2cb800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb800cbd0 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cb800eed0 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.609+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb4009d00 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.610+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8003e70 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.610+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc074210 con 0x7f2cbc102bd0 2024-03-16T20:33:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.611+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc10a620 con 0x7f2cbc102bd0 2024-03-16T20:33:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.612+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2cb8004000 con 0x7f2cbc102bd0 2024-03-16T20:33:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.613+0000 7f2caf7fe700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 0x7f2ca80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:33:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.613+0000 7f2cc230f700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 0x7f2ca80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:33:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.613+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2cb8014070 con 0x7f2cbc102bd0 2024-03-16T20:33:58.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.613+0000 7f2cc230f700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 0x7f2ca80a3f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0005950 tx=0x7f2cb000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:33:58.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.616+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2cb80969c0 con 0x7f2cbc102bd0 2024-03-16T20:33:58.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.869+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2cbc068ca0 con 0x7f2cbc102bd0 2024-03-16T20:33:58.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.869+0000 7f2caf7fe700 1 -- 172.21.15.23:0/3824747760 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f2cb809a080 con 0x7f2cbc102bd0 2024-03-16T20:33:58.875 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 msgr2=0x7f2ca80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 0x7f2ca80a3f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0005950 tx=0x7f2cb000a300 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 msgr2=0x7f2cbc198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800ea30 tx=0x7f2cb800ed40 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 shutdown_connections 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ca80a1ac0 0x7f2ca80a3f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cbc107300 0x7f2cbc19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cbc102220 0x7f2cbc198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 --2- 172.21.15.23:0/3824747760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cbc102bd0 0x7f2cbc198b90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:33:58.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 >> 172.21.15.23:0/3824747760 conn(0x7f2cbc0fda90 msgr2=0x7f2cbc0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:33:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 shutdown_connections 2024-03-16T20:33:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:33:58.871+0000 7f2cc4573700 1 -- 172.21.15.23:0/3824747760 wait complete. 2024-03-16T20:33:59.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:58 smithi023 ceph-mon[31879]: pgmap v294: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:33:59.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:33:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3824747760' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:33:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:58 smithi031 ceph-mon[36281]: pgmap v294: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:33:59.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:33:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3824747760' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:33:59.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:58 smithi189 ceph-mon[36655]: pgmap v294: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:33:59.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:33:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3824747760' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:33:59.491 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-16T20:33:59.492 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T20:33:59.492 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T20:33:59.492 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T20:33:59.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:33:59.492 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-16T20:33:59.508 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T20:33:59.508 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T20:33:59.508 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T20:33:59.508 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42235sec preferred_lft 42235sec 2024-03-16T20:33:59.509 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-16T20:33:59.510 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:33:59.510 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:33:59.510 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-16T20:33:59.510 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-16T20:33:59.510 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T20:33:59.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:33:59.510 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T20:33:59.511 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:33:59.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:33:59.676 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T20:33:59.703 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-16T20:33:59.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:33:59.704 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-16T20:33:59.771 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-16T20:33:59.771 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T20:33:59.771 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-16T20:33:59.771 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-16T20:33:59.771 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T20:33:59.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-16T20:33:59.772 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:34:00.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.015 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-16T20:34:00.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.030 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-16T20:34:00.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-16T20:34:00.085 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:34:00.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:00 smithi023 ceph-mon[31879]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-16T20:34:00.133 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-16T20:34:00.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-16T20:34:00.133 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:34:00.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.285 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:34:00.285 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:34:00.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-16T20:34:00.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T20:34:00.286 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:34:00.329 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-16T20:34:00.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T20:34:00.329 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:00.383 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-16T20:34:00.384 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T20:34:00.384 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:00.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-16T20:34:00.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:00 smithi031 ceph-mon[36281]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-16T20:34:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:00 smithi189 ceph-mon[36655]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-16T20:34:00.508 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T20:34:00.563 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-16T20:34:00.564 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ca6b13104146c7dd17f67459f0380f578c5fbc085372aca2c0cb85510c2c969b/merged 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d7459882c02fce5b0dd47a1dde24eca1e1eafa4ff3712b5cb0c973f7aefaacef/merged 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/02ae4a08f41145ad6b8437a7acfd434ad22258776d518f69fce818a91c9266b3/merged 2024-03-16T20:34:00.565 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/b98ee3ac789d478d4138e271c5743be917b4179b90434cf863b3c15620d47b4f/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/33a6b1ecdce2fda13ef6f2e4caf1e5deb1dce296fba81a180dc564d675daf5a1/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/b2eb632d62bacca238efed4e6900bea337b46cd9b71b707e31dabc02b25c10c5/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/0a05a55886f2d3cf8bff7ae32271214dd3e9c5d3b05c8301a2e3be33ce68ba20/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/14e737d0c6862c86e682e18b290a39a23c57eda2a27776d9b89a25bb9f51aa13/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ecac8f7ab38bdfd1affad644a7c17d8901d6729a1a1d3387f824f024bbc52bfe/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/99d68c962cb012943ff6285b6b8fc079a0c49d7eb8fc96a4b2436a63b684d41a/merged 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:34:00.566 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:34:00.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.569 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-16T20:34:00.618 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-16T20:34:00.618 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-16T20:34:00.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-16T20:34:00.687 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-16T20:34:00.744 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-16T20:34:00.745 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ca6b13104146c7dd17f67459f0380f578c5fbc085372aca2c0cb85510c2c969b/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d7459882c02fce5b0dd47a1dde24eca1e1eafa4ff3712b5cb0c973f7aefaacef/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/02ae4a08f41145ad6b8437a7acfd434ad22258776d518f69fce818a91c9266b3/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/b98ee3ac789d478d4138e271c5743be917b4179b90434cf863b3c15620d47b4f/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/33a6b1ecdce2fda13ef6f2e4caf1e5deb1dce296fba81a180dc564d675daf5a1/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/b2eb632d62bacca238efed4e6900bea337b46cd9b71b707e31dabc02b25c10c5/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/0a05a55886f2d3cf8bff7ae32271214dd3e9c5d3b05c8301a2e3be33ce68ba20/merged 2024-03-16T20:34:00.746 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/14e737d0c6862c86e682e18b290a39a23c57eda2a27776d9b89a25bb9f51aa13/merged 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ecac8f7ab38bdfd1affad644a7c17d8901d6729a1a1d3387f824f024bbc52bfe/merged 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/99d68c962cb012943ff6285b6b8fc079a0c49d7eb8fc96a4b2436a63b684d41a/merged 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:34:00.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.747 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-16T20:34:00.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2024-03-16T20:34:00.749+0000 7f544acf6480 -1 init, newargv = 0x55560d00ad30 newargc=15 2024-03-16T20:34:00.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[72908]: starting ceph client 2024-03-16T20:34:00.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[72908]: starting fuse 2024-03-16T20:34:00.770 INFO:teuthology.orchestra.run.smithi023.stdout:119 2024-03-16T20:34:00.771 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-16T20:34:00.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo stdin-killer -- python3 -c ' 2024-03-16T20:34:00.773 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2024-03-16T20:34:00.773 DEBUG:teuthology.orchestra.run.smithi023:> import re 2024-03-16T20:34:00.773 DEBUG:teuthology.orchestra.run.smithi023:> import os 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-16T20:34:00.774 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> return f 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2024-03-16T20:34:00.775 DEBUG:teuthology.orchestra.run.smithi023:> ' 2024-03-16T20:34:00.892 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.72908.asok 2024-03-16T20:34:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-16T20:34:00.903 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72908.asok 2024-03-16T20:34:00.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:00.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72908.asok status 2024-03-16T20:34:01.041 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1681-gaba3c1d2 (aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b) squid (dev)", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "72908", 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14793, 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2024-03-16T20:34:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 14793 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2447989594 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2447989594 2024-03-16T20:34:01.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.14793 192.168.0.1:0/2447989594", 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "192.168.0.1:0/2447989594", 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 17, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 90, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2024-03-16T20:34:01.044 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:34:01.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:01.050 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2024-03-16T20:34:01.050 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:01.065 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2024-03-16T20:34:01.065 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:01.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:01.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:01.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:01.133 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-16T20:34:01.208 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2024-03-16T20:34:01.208 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-16T20:34:01.208 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:01.208 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:34:01.208 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T20:34:01.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:34:01.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/2152038132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 msgr2=0x7f3cbc101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/2152038132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc101930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009ca0 tx=0x7f3cb402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/2152038132 shutdown_connections 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/2152038132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 0x7f3cbc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/2152038132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc102700 0x7f3cbc102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/2152038132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc101930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/2152038132 >> 172.21.15.23:0/2152038132 conn(0x7f3cbc0fcaa0 msgr2=0x7f3cbc0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/2152038132 shutdown_connections 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.401+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/2152038132 wait complete. 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 Processor -- start 2024-03-16T20:34:01.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 -- start start 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc102700 0x7f3cbc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 0x7f3cbc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc19f0c0 con 0x7f3cbc101510 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc19f230 con 0x7f3cbc1030c0 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc3b2c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc19f3f0 con 0x7f3cbc102700 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48838/0 (socket says 172.21.15.23:48838) 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 -- 172.21.15.23:0/3280029891 learned_addr learned my addr 172.21.15.23:0/3280029891 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc20c9640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 0x7f3cbc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 -- 172.21.15.23:0/3280029891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc102700 msgr2=0x7f3cbc197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc102700 0x7f3cbc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 -- 172.21.15.23:0/3280029891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 msgr2=0x7f3cbc19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 0x7f3cbc19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.402+0000 7f3cc18c8640 1 -- 172.21.15.23:0/3280029891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb4009900 con 0x7f3cbc101510 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cc18c8640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4037040 tx=0x7f3cb4034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4032eb0 con 0x7f3cbc101510 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cb4003e50 con 0x7f3cbc101510 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc19f670 con 0x7f3cbc101510 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4040470 con 0x7f3cbc101510 2024-03-16T20:34:01.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.403+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc19fb20 con 0x7f3cbc101510 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.404+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc108830 con 0x7f3cbc101510 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.405+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3cb400a040 con 0x7f3cbc101510 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.405+0000 7f3cbaffd640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 0x7f3ca40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.406+0000 7f3cc10c7640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 0x7f3ca40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.406+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3cb40f0940 con 0x7f3cbc101510 2024-03-16T20:34:01.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.406+0000 7f3cc10c7640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 0x7f3ca40a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005fd0 tx=0x7f3cb0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:01.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.409+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3cb40b9910 con 0x7f3cbc101510 2024-03-16T20:34:01.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.594+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cbc10cf20 con 0x7f3cbc101510 2024-03-16T20:34:01.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.595+0000 7f3cbaffd640 1 -- 172.21.15.23:0/3280029891 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f3cb40041b0 con 0x7f3cbc101510 2024-03-16T20:34:01.601 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:01.601 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 msgr2=0x7f3ca40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 0x7f3ca40a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005fd0 tx=0x7f3cb0005850 comp rx=0 tx=0).stop 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 msgr2=0x7f3cbc1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4037040 tx=0x7f3cb4034d40 comp rx=0 tx=0).stop 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 shutdown_connections 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca40a1a30 0x7f3ca40a3ef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc1030c0 0x7f3cbc19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc102700 0x7f3cbc197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 --2- 172.21.15.23:0/3280029891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc101510 0x7f3cbc1975d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 >> 172.21.15.23:0/3280029891 conn(0x7f3cbc0fcaa0 msgr2=0x7f3cbc105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 shutdown_connections 2024-03-16T20:34:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.598+0000 7f3cc3b2c640 1 -- 172.21.15.23:0/3280029891 wait complete. 2024-03-16T20:34:01.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:34:01.614 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:34:01.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.715+0000 7f143b2ff640 1 -- 172.21.15.23:0/1427944352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341026f0 msgr2=0x7f1434102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.715+0000 7f143b2ff640 1 --2- 172.21.15.23:0/1427944352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341026f0 0x7f1434102b70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1424009900 tx=0x7f142402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:01.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- 172.21.15.23:0/1427944352 shutdown_connections 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- 172.21.15.23:0/1427944352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341030b0 0x7f1434109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- 172.21.15.23:0/1427944352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341026f0 0x7f1434102b70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- 172.21.15.23:0/1427944352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 0x7f1434101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- 172.21.15.23:0/1427944352 >> 172.21.15.23:0/1427944352 conn(0x7f14340fca90 msgr2=0x7f14340feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- 172.21.15.23:0/1427944352 shutdown_connections 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- 172.21.15.23:0/1427944352 wait complete. 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 Processor -- start 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- start start 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 0x7f14340752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341026f0 0x7f14340738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1434074580 con 0x7f14341030b0 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340746f0 con 0x7f1434101500 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143b2ff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a1600 con 0x7f14341026f0 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.716+0000 7f143989c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48852/0 (socket says 172.21.15.23:48852) 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 -- 172.21.15.23:0/3513081406 learned_addr learned my addr 172.21.15.23:0/3513081406 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143889a640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341026f0 0x7f14340738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143909b640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 0x7f14340752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 -- 172.21.15.23:0/3513081406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341026f0 msgr2=0x7f14340738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341026f0 0x7f14340738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 -- 172.21.15.23:0/3513081406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 msgr2=0x7f14340752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 0x7f14340752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 -- 172.21.15.23:0/3513081406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424009560 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143989c640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f142c00eb40 tx=0x7f142c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c004b20 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14341a18b0 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f142c004c80 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c010690 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.717+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341a1d90 con 0x7f14341030b0 2024-03-16T20:34:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.718+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14341088f0 con 0x7f14341030b0 2024-03-16T20:34:01.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.719+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f142c005020 con 0x7f14341030b0 2024-03-16T20:34:01.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.720+0000 7f142a7fc640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 0x7f14200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:01.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.720+0000 7f143909b640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 0x7f14200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:01.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.720+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f142c014070 con 0x7f14341030b0 2024-03-16T20:34:01.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.720+0000 7f143909b640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 0x7f14200a3ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1430005fd0 tx=0x7f1430005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:01.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.724+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f142c096d80 con 0x7f14341030b0 2024-03-16T20:34:01.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.907+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f143402d060 con 0x7f14341030b0 2024-03-16T20:34:01.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.907+0000 7f142a7fc640 1 -- 172.21.15.23:0/3513081406 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f142c09a440 con 0x7f14341030b0 2024-03-16T20:34:01.913 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 msgr2=0x7f14200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 0x7f14200a3ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1430005fd0 tx=0x7f1430005850 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 msgr2=0x7f1434073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f142c00eb40 tx=0x7f142c00c2d0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 shutdown_connections 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14200a1a30 0x7f14200a3ef0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14341030b0 0x7f1434073e20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14341026f0 0x7f14340738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 --2- 172.21.15.23:0/3513081406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1434101500 0x7f14340752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 >> 172.21.15.23:0/3513081406 conn(0x7f14340fca90 msgr2=0x7f1434105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:01.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 shutdown_connections 2024-03-16T20:34:01.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:01.909+0000 7f143b2ff640 1 -- 172.21.15.23:0/3513081406 wait complete. 2024-03-16T20:34:01.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3280029891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:01.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:34:01.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3280029891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:01.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3280029891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 -- 172.21.15.23:0/1132413299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 msgr2=0x7f6874101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 --2- 172.21.15.23:0/1132413299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f6874101a60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f685c009900 tx=0x7f685c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 -- 172.21.15.23:0/1132413299 shutdown_connections 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 --2- 172.21.15.23:0/1132413299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 --2- 172.21.15.23:0/1132413299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f6874101a60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 --2- 172.21.15.23:0/1132413299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f6874100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 -- 172.21.15.23:0/1132413299 >> 172.21.15.23:0/1132413299 conn(0x7f68740fb980 msgr2=0x7f68740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 -- 172.21.15.23:0/1132413299 shutdown_connections 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.025+0000 7f6879676640 1 -- 172.21.15.23:0/1132413299 wait complete. 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 Processor -- start 2024-03-16T20:34:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 -- start start 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f687410acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874109830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6874109f90 con 0x7f68741015e0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687410a100 con 0x7f6874101fa0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6879676640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741a1640 con 0x7f68741003f0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f68727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f68727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48864/0 (socket says 172.21.15.23:48864) 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f68727fc640 1 -- 172.21.15.23:0/151099076 learned_addr learned my addr 172.21.15.23:0/151099076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f6872ffd640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f687410acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.026+0000 7f68737fe640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874109830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 -- 172.21.15.23:0/151099076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 msgr2=0x7f687410acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f687410acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 -- 172.21.15.23:0/151099076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 msgr2=0x7f6874109830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874109830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 -- 172.21.15.23:0/151099076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c009560 con 0x7f68741015e0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f6872ffd640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f687410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68737fe640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f68727fc640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f685c036040 tx=0x7f685c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c0043d0 con 0x7f68741015e0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f685c0052d0 con 0x7f68741015e0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68741a1890 con 0x7f68741015e0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c003cc0 con 0x7f68741015e0 2024-03-16T20:34:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.027+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68741a1da0 con 0x7f68741015e0 2024-03-16T20:34:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.028+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68741077e0 con 0x7f68741015e0 2024-03-16T20:34:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.029+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f685c005440 con 0x7f68741015e0 2024-03-16T20:34:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.029+0000 7f686bfff640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 0x7f68600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.030+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f685c0eff30 con 0x7f68741015e0 2024-03-16T20:34:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.030+0000 7f6872ffd640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 0x7f68600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.030+0000 7f6872ffd640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 0x7f68600a3ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6874101440 tx=0x7f686c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.033+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f685c0b8f00 con 0x7f68741015e0 2024-03-16T20:34:02.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.216+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f687410d150 con 0x7f68741015e0 2024-03-16T20:34:02.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.216+0000 7f686bfff640 1 -- 172.21.15.23:0/151099076 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f685c0bc5c0 con 0x7f68741015e0 2024-03-16T20:34:02.222 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 msgr2=0x7f68600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 0x7f68600a3ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6874101440 tx=0x7f686c005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 msgr2=0x7f68741092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f685c036040 tx=0x7f685c005e80 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 shutdown_connections 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f68600a1a30 0x7f68600a3ef0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874101fa0 0x7f6874109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68741015e0 0x7f68741092f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 --2- 172.21.15.23:0/151099076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68741003f0 0x7f687410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.218+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 >> 172.21.15.23:0/151099076 conn(0x7f68740fb980 msgr2=0x7f68741049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.219+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 shutdown_connections 2024-03-16T20:34:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.219+0000 7f6879676640 1 -- 172.21.15.23:0/151099076 wait complete. 2024-03-16T20:34:02.234 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 -- 172.21.15.23:0/1154237550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef81003f0 msgr2=0x7f1ef8100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/1154237550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef81003f0 0x7f1ef8100810 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009900 tx=0x7f1ee802bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 -- 172.21.15.23:0/1154237550 shutdown_connections 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/1154237550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef8101fa0 0x7f1ef8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/1154237550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81015e0 0x7f1ef8101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/1154237550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef81003f0 0x7f1ef8100810 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 -- 172.21.15.23:0/1154237550 >> 172.21.15.23:0/1154237550 conn(0x7f1ef80fb980 msgr2=0x7f1ef80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 -- 172.21.15.23:0/1154237550 shutdown_connections 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.334+0000 7f1efe9f4640 1 -- 172.21.15.23:0/1154237550 wait complete. 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 Processor -- start 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 -- start start 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 0x7f1ef81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef81015e0 0x7f1ef8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef819dfe0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef819e150 con 0x7f1ef81003f0 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efe9f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef819e310 con 0x7f1ef81015e0 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efcf91640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efcf91640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48872/0 (socket says 172.21.15.23:48872) 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1efcf91640 1 -- 172.21.15.23:0/3415658234 learned_addr learned my addr 172.21.15.23:0/3415658234 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.335+0000 7f1ef77fe640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef81015e0 0x7f1ef8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1ef7fff640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 0x7f1ef81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 -- 172.21.15.23:0/3415658234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef81015e0 msgr2=0x7f1ef8196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef81015e0 0x7f1ef8196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 -- 172.21.15.23:0/3415658234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 msgr2=0x7f1ef81964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 0x7f1ef81964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 -- 172.21.15.23:0/3415658234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8009560 con 0x7f1ef8101fa0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1ef7fff640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 0x7f1ef81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efcf91640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000eca0 tx=0x7f1ef000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0013070 con 0x7f1ef8101fa0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0005020 con 0x7f1ef8101fa0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef80686b0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef00099b0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.336+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef8068c10 con 0x7f1ef8101fa0 2024-03-16T20:34:02.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.337+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef80682e0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.338+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1ef0005520 con 0x7f1ef8101fa0 2024-03-16T20:34:02.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.338+0000 7f1ef57fa640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 0x7f1ee00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.339+0000 7f1ef7fff640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 0x7f1ee00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.339+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1ef00ceef0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.339+0000 7f1ef7fff640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 0x7f1ee00a3f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8036040 tx=0x7f1ee8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.343+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ef0097ec0 con 0x7f1ef8101fa0 2024-03-16T20:34:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.525+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ef8107360 con 0x7f1ef8101fa0 2024-03-16T20:34:02.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.526+0000 7f1ef57fa640 1 -- 172.21.15.23:0/3415658234 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ef009b580 con 0x7f1ef8101fa0 2024-03-16T20:34:02.531 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:34:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 msgr2=0x7f1ee00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 0x7f1ee00a3f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8036040 tx=0x7f1ee8000bc0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 msgr2=0x7f1ef819daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000eca0 tx=0x7f1ef000b690 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 shutdown_connections 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1ee00a1ab0 0x7f1ee00a3f70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ef8101fa0 0x7f1ef819daa0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef81015e0 0x7f1ef8196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 --2- 172.21.15.23:0/3415658234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ef81003f0 0x7f1ef81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 >> 172.21.15.23:0/3415658234 conn(0x7f1ef80fb980 msgr2=0x7f1ef8104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 shutdown_connections 2024-03-16T20:34:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.528+0000 7f1efe9f4640 1 -- 172.21.15.23:0/3415658234 wait complete. 2024-03-16T20:34:02.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 -- 172.21.15.23:0/4260041529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370069760 msgr2=0x7fc3701000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 --2- 172.21.15.23:0/4260041529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370069760 0x7fc3701000f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc360009900 tx=0x7fc36002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 -- 172.21.15.23:0/4260041529 shutdown_connections 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 --2- 172.21.15.23:0/4260041529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc370100d80 0x7fc370101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 --2- 172.21.15.23:0/4260041529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370069760 0x7fc3701000f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 --2- 172.21.15.23:0/4260041529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3700692b0 0x7fc3700ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 -- 172.21.15.23:0/4260041529 >> 172.21.15.23:0/4260041529 conn(0x7fc3700fb120 msgr2=0x7fc3700fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 -- 172.21.15.23:0/4260041529 shutdown_connections 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.643+0000 7fc37736c640 1 -- 172.21.15.23:0/4260041529 wait complete. 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 Processor -- start 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 -- start start 2024-03-16T20:34:02.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 0x7fc370102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 0x7fc37019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019f080 con 0x7fc370100d80 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019f1f0 con 0x7fc3700692b0 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc37736c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37019f3b0 con 0x7fc370069760 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375909640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375108640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 0x7fc370102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 0x7fc37019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375909640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48896/0 (socket says 172.21.15.23:48896) 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375909640 1 -- 172.21.15.23:0/3854797007 learned_addr learned my addr 172.21.15.23:0/3854797007 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375108640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 0x7fc370102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:54944/0 (socket says 172.21.15.23:54944) 2024-03-16T20:34:02.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 -- 172.21.15.23:0/3854797007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 msgr2=0x7fc370102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 0x7fc370102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 -- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 msgr2=0x7fc37019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 -- 172.21.15.23:0/3854797007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360009560 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc375909640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.644+0000 7fc374907640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 0x7fc37019c2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc360036040 tx=0x7fc360004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.645+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360004aa0 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.645+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37019f630 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.645+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3600052d0 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.645+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37019fb60 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.645+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360003d60 con 0x7fc370069760 2024-03-16T20:34:02.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.646+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370107910 con 0x7fc370069760 2024-03-16T20:34:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.647+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc360005440 con 0x7fc370069760 2024-03-16T20:34:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.647+0000 7fc3667fc640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 0x7fc35c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.647+0000 7fc375108640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 0x7fc35c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.647+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc3600f0670 con 0x7fc370069760 2024-03-16T20:34:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.648+0000 7fc375108640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 0x7fc35c0a3ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc370100be0 tx=0x7fc36c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.651+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc3600b9640 con 0x7fc370069760 2024-03-16T20:34:02.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.832+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc37010c080 con 0x7fc370069760 2024-03-16T20:34:02.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.832+0000 7fc3667fc640 1 -- 172.21.15.23:0/3854797007 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc3600bcd00 con 0x7fc370069760 2024-03-16T20:34:02.838 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 msgr2=0x7fc35c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 0x7fc35c0a3ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc370100be0 tx=0x7fc36c005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 msgr2=0x7fc37019c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 0x7fc37019c2b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc360036040 tx=0x7fc360004270 comp rx=0 tx=0).stop 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 shutdown_connections 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc35c0a1a30 0x7fc35c0a3ef0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc370100d80 0x7fc37019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc370069760 0x7fc37019c2b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 --2- 172.21.15.23:0/3854797007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3700692b0 0x7fc370102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 >> 172.21.15.23:0/3854797007 conn(0x7fc3700fb120 msgr2=0x7fc3700fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 shutdown_connections 2024-03-16T20:34:02.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.834+0000 7fc37736c640 1 -- 172.21.15.23:0/3854797007 wait complete. 2024-03-16T20:34:02.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3513081406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:02 smithi023 ceph-mon[31879]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:34:02.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/151099076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3415658234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T20:34:02.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3513081406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:02 smithi031 ceph-mon[36281]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:34:02.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/151099076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3415658234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 -- 172.21.15.23:0/1968458711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8107270 msgr2=0x7f10f8109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 --2- 172.21.15.23:0/1968458711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8107270 0x7f10f8109660 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009900 tx=0x7f10f002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 -- 172.21.15.23:0/1968458711 shutdown_connections 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 --2- 172.21.15.23:0/1968458711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8107270 0x7f10f8109660 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 --2- 172.21.15.23:0/1968458711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8104940 0x7f10f8106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 --2- 172.21.15.23:0/1968458711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8102010 0x7f10f8104400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.948+0000 7f10fe18c640 1 -- 172.21.15.23:0/1968458711 >> 172.21.15.23:0/1968458711 conn(0x7f10f80fb980 msgr2=0x7f10f80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- 172.21.15.23:0/1968458711 shutdown_connections 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- 172.21.15.23:0/1968458711 wait complete. 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 Processor -- start 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- start start 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 0x7f10f8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8107270 0x7f10f819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f819f070 con 0x7f10f8102010 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f819f1e0 con 0x7f10f8107270 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10fe18c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f819f3a0 con 0x7f10f8104940 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.949+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48926/0 (socket says 172.21.15.23:48926) 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 -- 172.21.15.23:0/2665276860 learned_addr learned my addr 172.21.15.23:0/2665276860 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:02.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f7fff640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8107270 0x7f10f819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f6ffd640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 0x7f10f8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 -- 172.21.15.23:0/2665276860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 msgr2=0x7f10f8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 0x7f10f8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 -- 172.21.15.23:0/2665276860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8107270 msgr2=0x7f10f819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8107270 0x7f10f819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 -- 172.21.15.23:0/2665276860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0009560 con 0x7f10f8102010 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f6ffd640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 0x7f10f8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f77fe640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10e800eb90 tx=0x7f10e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e80099b0 con 0x7f10f8102010 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e800ee90 con 0x7f10f8102010 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f819f680 con 0x7f10f8102010 2024-03-16T20:34:02.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e80107a0 con 0x7f10f8102010 2024-03-16T20:34:02.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.950+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f819fb40 con 0x7f10f8102010 2024-03-16T20:34:02.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.951+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80fd470 con 0x7f10f8102010 2024-03-16T20:34:02.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.952+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f10e8005020 con 0x7f10f8102010 2024-03-16T20:34:02.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.953+0000 7f10f4ff9640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 0x7f10e40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:02.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.953+0000 7f10f6ffd640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 0x7f10e40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:02.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.953+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f10e8014070 con 0x7f10f8102010 2024-03-16T20:34:02.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.953+0000 7f10f6ffd640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 0x7f10e40a3ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10f8198a90 tx=0x7f10e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:02.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:02.957+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f10e80d2070 con 0x7f10f8102010 2024-03-16T20:34:02.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3513081406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:02 smithi189 ceph-mon[36655]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-16T20:34:02.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/151099076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:02.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3415658234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:03.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.151+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f10f810cf20 con 0x7f10f8102010 2024-03-16T20:34:03.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.152+0000 7f10f4ff9640 1 -- 172.21.15.23:0/2665276860 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+592 (secure 0 0 0) 0x7f10e8096e80 con 0x7f10f8102010 2024-03-16T20:34:03.157 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:03.157 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1269138528 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4262074056 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2140563160 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4012523374 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1525116937 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3223781845 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-16T20:34:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 msgr2=0x7f10e40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 0x7f10e40a3ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f10f8198a90 tx=0x7f10e0005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 msgr2=0x7f10f8197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10e800eb90 tx=0x7f10e800c6a0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 shutdown_connections 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f10e40a1a30 0x7f10e40a3ef0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10f8107270 0x7f10f819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f8104940 0x7f10f8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 --2- 172.21.15.23:0/2665276860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f8102010 0x7f10f8197580 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 >> 172.21.15.23:0/2665276860 conn(0x7f10f80fb980 msgr2=0x7f10f8102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 shutdown_connections 2024-03-16T20:34:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.154+0000 7f10fe18c640 1 -- 172.21.15.23:0/2665276860 wait complete. 2024-03-16T20:34:03.170 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:34:03.170 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T20:34:03.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:34:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 -- 172.21.15.23:0/443730469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101fc0 msgr2=0x7f0f28108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 --2- 172.21.15.23:0/443730469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101fc0 0x7f0f28108850 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009ca0 tx=0x7f0f2002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 -- 172.21.15.23:0/443730469 shutdown_connections 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 --2- 172.21.15.23:0/443730469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101fc0 0x7f0f28108850 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 --2- 172.21.15.23:0/443730469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28101600 0x7f0f28101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 --2- 172.21.15.23:0/443730469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28100410 0x7f0f28100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 -- 172.21.15.23:0/443730469 >> 172.21.15.23:0/443730469 conn(0x7f0f280fb980 msgr2=0x7f0f280fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 -- 172.21.15.23:0/443730469 shutdown_connections 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.322+0000 7f0f2d247640 1 -- 172.21.15.23:0/443730469 wait complete. 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 Processor -- start 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 -- start start 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 0x7f0f280691e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28101fc0 0x7f0f2819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819f010 con 0x7f0f28101600 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819f180 con 0x7f0f28100410 2024-03-16T20:34:03.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.323+0000 7f0f2d247640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819f340 con 0x7f0f28101fc0 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48952/0 (socket says 172.21.15.23:48952) 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2659c640 1 -- 172.21.15.23:0/1430213100 learned_addr learned my addr 172.21.15.23:0/1430213100 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 0x7f0f280691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2759e640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28101fc0 0x7f0f2819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 -- 172.21.15.23:0/1430213100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28101fc0 msgr2=0x7f0f2819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28101fc0 0x7f0f2819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 -- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 msgr2=0x7f0f2819c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 -- 172.21.15.23:0/1430213100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20009900 con 0x7f0f28100410 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2659c640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f26d9d640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 0x7f0f280691e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800d6a0 tx=0x7f0f1800db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18004220 con 0x7f0f28100410 2024-03-16T20:34:03.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2819f620 con 0x7f0f28100410 2024-03-16T20:34:03.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f18004380 con 0x7f0f28100410 2024-03-16T20:34:03.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18003d80 con 0x7f0f28100410 2024-03-16T20:34:03.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.324+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2819fb80 con 0x7f0f28100410 2024-03-16T20:34:03.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.325+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f28107730 con 0x7f0f28100410 2024-03-16T20:34:03.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.326+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0f18009820 con 0x7f0f28100410 2024-03-16T20:34:03.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.327+0000 7f0f1ffff640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 0x7f0f140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.327+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0f180ccef0 con 0x7f0f28100410 2024-03-16T20:34:03.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.327+0000 7f0f2659c640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 0x7f0f140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.327+0000 7f0f2659c640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 0x7f0f140a3ef0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0f28068f50 tx=0x7f0f10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.331+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0f18095f70 con 0x7f0f28100410 2024-03-16T20:34:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.513+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f2810bfa0 con 0x7f0f28100410 2024-03-16T20:34:03.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.514+0000 7f0f1ffff640 1 -- 172.21.15.23:0/1430213100 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f0f2810bfa0 con 0x7f0f28100410 2024-03-16T20:34:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:03.520 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 msgr2=0x7f0f140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 0x7f0f140a3ef0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0f28068f50 tx=0x7f0f10005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 msgr2=0x7f0f280691e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 0x7f0f280691e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800d6a0 tx=0x7f0f1800db60 comp rx=0 tx=0).stop 2024-03-16T20:34:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 shutdown_connections 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0f140a1a30 0x7f0f140a3ef0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f28101fc0 0x7f0f2819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f28101600 0x7f0f2819c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 --2- 172.21.15.23:0/1430213100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f28100410 0x7f0f280691e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 >> 172.21.15.23:0/1430213100 conn(0x7f0f280fb980 msgr2=0x7f0f28104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 shutdown_connections 2024-03-16T20:34:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.517+0000 7f0f2d247640 1 -- 172.21.15.23:0/1430213100 wait complete. 2024-03-16T20:34:03.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:34:03.533 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:34:03.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 -- 172.21.15.23:0/1923857995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101030 msgr2=0x7f11c41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 --2- 172.21.15.23:0/1923857995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101030 0x7f11c41088d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009ca0 tx=0x7f11bc02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 -- 172.21.15.23:0/1923857995 shutdown_connections 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 --2- 172.21.15.23:0/1923857995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101030 0x7f11c41088d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 --2- 172.21.15.23:0/1923857995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 0x7f11c4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 --2- 172.21.15.23:0/1923857995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101a20 0x7f11c4101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 -- 172.21.15.23:0/1923857995 >> 172.21.15.23:0/1923857995 conn(0x7f11c40fb100 msgr2=0x7f11c40fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 -- 172.21.15.23:0/1923857995 shutdown_connections 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.633+0000 7f11cac4d640 1 -- 172.21.15.23:0/1923857995 wait complete. 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 Processor -- start 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 -- start start 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 0x7f11c4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101030 0x7f11c4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419f070 con 0x7f11c4101a20 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419f1e0 con 0x7f11c4101030 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11cac4d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419f3a0 con 0x7f11c4100670 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48966/0 (socket says 172.21.15.23:48966) 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 -- 172.21.15.23:0/3464776755 learned_addr learned my addr 172.21.15.23:0/3464776755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c89e9640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 0x7f11c4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c3fff640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101030 0x7f11c4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 -- 172.21.15.23:0/3464776755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 msgr2=0x7f11c4197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 0x7f11c4197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 -- 172.21.15.23:0/3464776755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101030 msgr2=0x7f11c4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101030 0x7f11c4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 -- 172.21.15.23:0/3464776755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009900 con 0x7f11c4101a20 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.634+0000 7f11c91ea640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11bc037040 tx=0x7f11bc032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.635+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc034c90 con 0x7f11c4101a20 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.635+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11bc034df0 con 0x7f11c4101a20 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.635+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c419f620 con 0x7f11c4101a20 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.635+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc004890 con 0x7f11c4101a20 2024-03-16T20:34:03.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.635+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c419fb20 con 0x7f11c4101a20 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.636+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c410be60 con 0x7f11c4101a20 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.637+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f11bc00a040 con 0x7f11c4101a20 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.637+0000 7f11c1ffb640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 0x7f11b40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.637+0000 7f11c89e9640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 0x7f11b40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.637+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f11bc0f0330 con 0x7f11c4101a20 2024-03-16T20:34:03.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.638+0000 7f11c89e9640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 0x7f11b40a3f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11c40698c0 tx=0x7f11b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.642+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f11bc0b9300 con 0x7f11c4101a20 2024-03-16T20:34:03.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.825+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11c419ffe0 con 0x7f11c4101a20 2024-03-16T20:34:03.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.826+0000 7f11c1ffb640 1 -- 172.21.15.23:0/3464776755 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11bc0bc9c0 con 0x7f11c4101a20 2024-03-16T20:34:03.831 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:34:03.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 msgr2=0x7f11b40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 0x7f11b40a3f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f11c40698c0 tx=0x7f11b8005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:03.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 msgr2=0x7f11c419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11bc037040 tx=0x7f11bc032fc0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 shutdown_connections 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f11b40a1ab0 0x7f11b40a3f70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c4101a20 0x7f11c419eb30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11c4101030 0x7f11c4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 --2- 172.21.15.23:0/3464776755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11c4100670 0x7f11c4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 >> 172.21.15.23:0/3464776755 conn(0x7f11c40fb100 msgr2=0x7f11c40fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 shutdown_connections 2024-03-16T20:34:03.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.828+0000 7f11cac4d640 1 -- 172.21.15.23:0/3464776755 wait complete. 2024-03-16T20:34:03.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3854797007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:03.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2665276860' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:03.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1430213100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:03.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:34:03.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:03 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3854797007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:03.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:03 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2665276860' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:03.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:03 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1430213100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:03.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- 172.21.15.23:0/2053624508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 msgr2=0x7fb78c1043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 --2- 172.21.15.23:0/2053624508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c1043e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb774009900 tx=0x7fb77402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- 172.21.15.23:0/2053624508 shutdown_connections 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 --2- 172.21.15.23:0/2053624508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 0x7fb78c109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 --2- 172.21.15.23:0/2053624508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 0x7fb78c106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 --2- 172.21.15.23:0/2053624508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c1043e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- 172.21.15.23:0/2053624508 >> 172.21.15.23:0/2053624508 conn(0x7fb78c0fb9a0 msgr2=0x7fb78c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- 172.21.15.23:0/2053624508 shutdown_connections 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- 172.21.15.23:0/2053624508 wait complete. 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 Processor -- start 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.944+0000 7fb791956640 1 -- start start 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 0x7fb78c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 0x7fb78c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1a1320 con 0x7fb78c101ff0 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1a1490 con 0x7fb78c107250 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1a1650 con 0x7fb78c104920 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48984/0 (socket says 172.21.15.23:48984) 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 -- 172.21.15.23:0/4184684561 learned_addr learned my addr 172.21.15.23:0/4184684561 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78b7fe640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 0x7fb78c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78a7fc640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 0x7fb78c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 -- 172.21.15.23:0/4184684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 msgr2=0x7fb78c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 0x7fb78c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 -- 172.21.15.23:0/4184684561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 msgr2=0x7fb78c1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 0x7fb78c1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 -- 172.21.15.23:0/4184684561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb774009560 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb78affd640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb774006010 tx=0x7fb774005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7740043d0 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7740052d0 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c1a18d0 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774003cc0 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.945+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c1a1e30 con 0x7fb78c101ff0 2024-03-16T20:34:03.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.946+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c108d40 con 0x7fb78c101ff0 2024-03-16T20:34:03.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.947+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb774005440 con 0x7fb78c101ff0 2024-03-16T20:34:03.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.948+0000 7fb790954640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 0x7fb7780a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:03.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.948+0000 7fb78a7fc640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 0x7fb7780a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:03.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.948+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb7740ef7d0 con 0x7fb78c101ff0 2024-03-16T20:34:03.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.948+0000 7fb78a7fc640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 0x7fb7780a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb78c19ad40 tx=0x7fb77c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:03.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:03.953+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb7740b87a0 con 0x7fb78c101ff0 2024-03-16T20:34:03.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:03 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3854797007' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:03.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:03 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2665276860' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:03.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:03 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1430213100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.132+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb78c02d000 con 0x7fb78c101ff0 2024-03-16T20:34:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.132+0000 7fb790954640 1 -- 172.21.15.23:0/4184684561 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7740bbe60 con 0x7fb78c101ff0 2024-03-16T20:34:04.138 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:34:04.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 msgr2=0x7fb7780a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 0x7fb7780a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb78c19ad40 tx=0x7fb77c005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:04.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 msgr2=0x7fb78c199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb774006010 tx=0x7fb774005e80 comp rx=0 tx=0).stop 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 shutdown_connections 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb7780a1ab0 0x7fb7780a3f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb78c107250 0x7fb78c1a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb78c104920 0x7fb78c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 --2- 172.21.15.23:0/4184684561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb78c101ff0 0x7fb78c199830 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 >> 172.21.15.23:0/4184684561 conn(0x7fb78c0fb9a0 msgr2=0x7fb78c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 shutdown_connections 2024-03-16T20:34:04.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.134+0000 7fb791956640 1 -- 172.21.15.23:0/4184684561 wait complete. 2024-03-16T20:34:04.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:34:04.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- 172.21.15.23:0/145667259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c101fc0 msgr2=0x7f088c108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 --2- 172.21.15.23:0/145667259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c101fc0 0x7f088c108850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0884009ca0 tx=0x7f088402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- 172.21.15.23:0/145667259 shutdown_connections 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 --2- 172.21.15.23:0/145667259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c101fc0 0x7f088c108850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 --2- 172.21.15.23:0/145667259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101600 0x7f088c101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 --2- 172.21.15.23:0/145667259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c100410 0x7f088c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- 172.21.15.23:0/145667259 >> 172.21.15.23:0/145667259 conn(0x7f088c0fb9a0 msgr2=0x7f088c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- 172.21.15.23:0/145667259 shutdown_connections 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- 172.21.15.23:0/145667259 wait complete. 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 Processor -- start 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.250+0000 7f0892b77640 1 -- start start 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c101600 0x7f088c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 0x7f088c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c19f000 con 0x7f088c100410 2024-03-16T20:34:04.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c19f170 con 0x7f088c101fc0 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c19f330 con 0x7f088c101600 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49008/0 (socket says 172.21.15.23:49008) 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 -- 172.21.15.23:0/1318359822 learned_addr learned my addr 172.21.15.23:0/1318359822 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f088bfff640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c101600 0x7f088c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0891114640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 0x7f088c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 -- 172.21.15.23:0/1318359822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c101600 msgr2=0x7f088c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c101600 0x7f088c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 -- 172.21.15.23:0/1318359822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 msgr2=0x7f088c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 0x7f088c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 -- 172.21.15.23:0/1318359822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884009900 con 0x7f088c100410 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0891114640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 0x7f088c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:04.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0890913640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f088000e9c0 tx=0x7f088000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880009700 con 0x7f088c100410 2024-03-16T20:34:04.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c19f610 con 0x7f088c100410 2024-03-16T20:34:04.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f088000cd80 con 0x7f088c100410 2024-03-16T20:34:04.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.251+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880010430 con 0x7f088c100410 2024-03-16T20:34:04.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.252+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c19fb20 con 0x7f088c100410 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.253+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c107730 con 0x7f088c100410 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.253+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0880003680 con 0x7f088c100410 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.254+0000 7f0889ffb640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 0x7f087c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.254+0000 7f088bfff640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 0x7f087c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.254+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0880014070 con 0x7f088c100410 2024-03-16T20:34:04.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.255+0000 7f088bfff640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 0x7f087c0a3ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f088c198a20 tx=0x7f087800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.259+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0880096a70 con 0x7f088c100410 2024-03-16T20:34:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.442+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f088c10cf20 con 0x7f088c100410 2024-03-16T20:34:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.442+0000 7f0889ffb640 1 -- 172.21.15.23:0/1318359822 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f088009a130 con 0x7f088c100410 2024-03-16T20:34:04.447 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:34:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 msgr2=0x7f087c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 0x7f087c0a3ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f088c198a20 tx=0x7f087800b390 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 msgr2=0x7f088c197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f088000e9c0 tx=0x7f088000ee80 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 shutdown_connections 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f087c0a1a30 0x7f087c0a3ef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f088c101fc0 0x7f088c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f088c101600 0x7f088c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 --2- 172.21.15.23:0/1318359822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c100410 0x7f088c197510 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 >> 172.21.15.23:0/1318359822 conn(0x7f088c0fb9a0 msgr2=0x7f088c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 shutdown_connections 2024-03-16T20:34:04.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.444+0000 7f0892b77640 1 -- 172.21.15.23:0/1318359822 wait complete. 2024-03-16T20:34:04.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:34:04.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 -- 172.21.15.23:0/1854795620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63181003f0 msgr2=0x7f6318100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 --2- 172.21.15.23:0/1854795620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63181003f0 0x7f6318100810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6300009900 tx=0x7f630002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 -- 172.21.15.23:0/1854795620 shutdown_connections 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 --2- 172.21.15.23:0/1854795620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6318101fa0 0x7f6318108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 --2- 172.21.15.23:0/1854795620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181015e0 0x7f6318101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 --2- 172.21.15.23:0/1854795620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63181003f0 0x7f6318100810 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.559+0000 7f631d807640 1 -- 172.21.15.23:0/1854795620 >> 172.21.15.23:0/1854795620 conn(0x7f63180fb980 msgr2=0x7f63180fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- 172.21.15.23:0/1854795620 shutdown_connections 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- 172.21.15.23:0/1854795620 wait complete. 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 Processor -- start 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- start start 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181003f0 0x7f6318197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63181015e0 0x7f6318197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631819f0f0 con 0x7f6318101fa0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631819f260 con 0x7f63181015e0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.560+0000 7f631d807640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631819f420 con 0x7f63181003f0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49024/0 (socket says 172.21.15.23:49024) 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f6316ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181003f0 0x7f6318197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 -- 172.21.15.23:0/3746022960 learned_addr learned my addr 172.21.15.23:0/3746022960 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63167fc640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63181015e0 0x7f6318197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 -- 172.21.15.23:0/3746022960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181003f0 msgr2=0x7f6318197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181003f0 0x7f6318197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 -- 172.21.15.23:0/3746022960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63181015e0 msgr2=0x7f6318197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63181015e0 0x7f6318197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 -- 172.21.15.23:0/3746022960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300009560 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f63177fe640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f631000eaa0 tx=0x7f631000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631000cdb0 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6310005020 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631819f700 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63100116b0 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.561+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631819fbc0 con 0x7f6318101fa0 2024-03-16T20:34:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.562+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6318107830 con 0x7f6318101fa0 2024-03-16T20:34:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.563+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6310005520 con 0x7f6318101fa0 2024-03-16T20:34:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.564+0000 7f631c805640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 0x7f63040a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.564+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f63100cd7c0 con 0x7f6318101fa0 2024-03-16T20:34:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.564+0000 7f6316ffd640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 0x7f63040a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.564+0000 7f6316ffd640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 0x7f63040a3ef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6300005ab0 tx=0x7f6300005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.568+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6310096840 con 0x7f6318101fa0 2024-03-16T20:34:04.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.751+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f631802cf60 con 0x7f6318101fa0 2024-03-16T20:34:04.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.752+0000 7f631c805640 1 -- 172.21.15.23:0/3746022960 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6310099f00 con 0x7f6318101fa0 2024-03-16T20:34:04.757 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.753+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 msgr2=0x7f63040a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.753+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 0x7f63040a3ef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6300005ab0 tx=0x7f6300005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.753+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 msgr2=0x7f631819ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.753+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f631000eaa0 tx=0x7f631000ef60 comp rx=0 tx=0).stop 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 shutdown_connections 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f63040a1a30 0x7f63040a3ef0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6318101fa0 0x7f631819ebb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63181015e0 0x7f6318197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 --2- 172.21.15.23:0/3746022960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63181003f0 0x7f6318197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 >> 172.21.15.23:0/3746022960 conn(0x7f63180fb980 msgr2=0x7f6318104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 shutdown_connections 2024-03-16T20:34:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.754+0000 7f631d807640 1 -- 172.21.15.23:0/3746022960 wait complete. 2024-03-16T20:34:04.768 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:04 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3464776755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:04 smithi023 ceph-mon[31879]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:04.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:04 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4184684561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:04 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1318359822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T20:34:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 -- 172.21.15.23:0/3676837016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 msgr2=0x7fc218100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/3676837016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc218100830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc210009900 tx=0x7fc21002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 -- 172.21.15.23:0/3676837016 shutdown_connections 2024-03-16T20:34:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/3676837016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 0x7fc218108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/3676837016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc218101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/3676837016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc218100830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 -- 172.21.15.23:0/3676837016 >> 172.21.15.23:0/3676837016 conn(0x7fc2180fb9a0 msgr2=0x7fc2180fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 -- 172.21.15.23:0/3676837016 shutdown_connections 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.868+0000 7fc21cdf4640 1 -- 172.21.15.23:0/3676837016 wait complete. 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 Processor -- start 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 -- start start 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc2181029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 0x7fc218102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc218103680 con 0x7fc218100410 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2181037f0 con 0x7fc218101600 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21cdf4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2181a1680 con 0x7fc218101fc0 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49046/0 (socket says 172.21.15.23:49046) 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.869+0000 7fc21659c640 1 -- 172.21.15.23:0/189970634 learned_addr learned my addr 172.21.15.23:0/189970634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc216d9d640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 0x7fc218102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc215d9b640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc2181029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 -- 172.21.15.23:0/189970634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 msgr2=0x7fc218102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 0x7fc218102f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 -- 172.21.15.23:0/189970634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 msgr2=0x7fc2181029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc2181029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 -- 172.21.15.23:0/189970634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc210009560 con 0x7fc218100410 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc215d9b640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc2181029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21659c640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc210036040 tx=0x7fc210004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210004df0 con 0x7fc218100410 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2181a18d0 con 0x7fc218100410 2024-03-16T20:34:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc210005430 con 0x7fc218100410 2024-03-16T20:34:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21003e450 con 0x7fc218100410 2024-03-16T20:34:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.870+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2181a1de0 con 0x7fc218100410 2024-03-16T20:34:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.871+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21810bdf0 con 0x7fc218100410 2024-03-16T20:34:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.872+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc210003680 con 0x7fc218100410 2024-03-16T20:34:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.873+0000 7fc20f7fe640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 0x7fc2040a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.873+0000 7fc215d9b640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 0x7fc2040a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.873+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc2100efb40 con 0x7fc218100410 2024-03-16T20:34:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.873+0000 7fc215d9b640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 0x7fc2040a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc218104110 tx=0x7fc200005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:04.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:04.877+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc2100b8b10 con 0x7fc218100410 2024-03-16T20:34:04.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:04 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3464776755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:04 smithi031 ceph-mon[36281]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:04.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:04 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4184684561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:04 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1318359822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:04 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3464776755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:04 smithi189 ceph-mon[36655]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:04.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:04 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4184684561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:34:04.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:04 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1318359822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:34:05.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.071+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc21810bfb0 con 0x7fc218100410 2024-03-16T20:34:05.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.072+0000 7fc20f7fe640 1 -- 172.21.15.23:0/189970634 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+592 (secure 0 0 0) 0x7fc2100bc1d0 con 0x7fc218100410 2024-03-16T20:34:05.077 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1269138528 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4262074056 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2140563160 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4012523374 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1525116937 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3223781845 2024-03-17T20:26:23.333931+0000 2024-03-16T20:34:05.078 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:34:05.079 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-16T20:34:05.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 msgr2=0x7fc2040a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 0x7fc2040a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc218104110 tx=0x7fc200005e50 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 msgr2=0x7fc2181043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc210036040 tx=0x7fc210004270 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 shutdown_connections 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc2040a1a30 0x7fc2040a3ef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc218101fc0 0x7fc218102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101600 0x7fc2181029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 --2- 172.21.15.23:0/189970634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc218100410 0x7fc2181043a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 >> 172.21.15.23:0/189970634 conn(0x7fc2180fb9a0 msgr2=0x7fc218069ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 shutdown_connections 2024-03-16T20:34:05.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.074+0000 7fc21cdf4640 1 -- 172.21.15.23:0/189970634 wait complete. 2024-03-16T20:34:05.090 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2024-03-16T20:34:05.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:05.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-16T20:34:05.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[72908]: fuse finished with error 0 and tester_r 0 2024-03-16T20:34:05.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-16T20:34:05.142 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:34:05.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:34:05.142 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-16T20:34:05.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 -- 172.21.15.23:0/196153693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101600 msgr2=0x7efdac101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 --2- 172.21.15.23:0/196153693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101600 0x7efdac101a80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efda0009900 tx=0x7efda002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 -- 172.21.15.23:0/196153693 shutdown_connections 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 --2- 172.21.15.23:0/196153693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101fc0 0x7efdac108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 --2- 172.21.15.23:0/196153693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101600 0x7efdac101a80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 --2- 172.21.15.23:0/196153693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 0x7efdac100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 -- 172.21.15.23:0/196153693 >> 172.21.15.23:0/196153693 conn(0x7efdac0fb980 msgr2=0x7efdac0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 -- 172.21.15.23:0/196153693 shutdown_connections 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.346+0000 7efdb36df640 1 -- 172.21.15.23:0/196153693 wait complete. 2024-03-16T20:34:05.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 Processor -- start 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 -- start start 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 0x7efdac1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 0x7efdac197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac19f090 con 0x7efdac101fc0 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac19f200 con 0x7efdac101600 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb36df640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac19f3c0 con 0x7efdac100410 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49062/0 (socket says 172.21.15.23:49062) 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb147b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 0x7efdac1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 -- 172.21.15.23:0/1999739646 learned_addr learned my addr 172.21.15.23:0/1999739646 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:05.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb0c7a640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 0x7efdac197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 -- 172.21.15.23:0/1999739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 msgr2=0x7efdac1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 0x7efdac1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 -- 172.21.15.23:0/1999739646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 msgr2=0x7efdac197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 0x7efdac197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb1c7c640 1 -- 172.21.15.23:0/1999739646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0009560 con 0x7efdac101fc0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.347+0000 7efdb0c7a640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 0x7efdac197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdb1c7c640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00eaa0 tx=0x7efd9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00cdb0 con 0x7efdac101fc0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd9c005020 con 0x7efdac101fc0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdac19f6a0 con 0x7efdac101fc0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c0116b0 con 0x7efdac101fc0 2024-03-16T20:34:05.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.348+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdac19fbb0 con 0x7efdac101fc0 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.349+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdac107800 con 0x7efdac101fc0 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.350+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efd9c005520 con 0x7efdac101fc0 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.351+0000 7efdaa7fc640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 0x7efd940a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.351+0000 7efdb147b640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 0x7efd940a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.351+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7efd9c0cd7f0 con 0x7efdac101fc0 2024-03-16T20:34:05.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.351+0000 7efdb147b640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 0x7efd940a3f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efdac101460 tx=0x7efda4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.355+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efd9c0967c0 con 0x7efdac101fc0 2024-03-16T20:34:05.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.538+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdac10cf20 con 0x7efdac101fc0 2024-03-16T20:34:05.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.539+0000 7efdaa7fc640 1 -- 172.21.15.23:0/1999739646 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7efd9c099e80 con 0x7efdac101fc0 2024-03-16T20:34:05.545 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:05.545 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 msgr2=0x7efd940a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 0x7efd940a3f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efdac101460 tx=0x7efda4005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 msgr2=0x7efdac19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00eaa0 tx=0x7efd9c00ef60 comp rx=0 tx=0).stop 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 shutdown_connections 2024-03-16T20:34:05.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efd940a1ab0 0x7efd940a3f70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdac101fc0 0x7efdac19eb50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac101600 0x7efdac197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 --2- 172.21.15.23:0/1999739646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efdac100410 0x7efdac1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 >> 172.21.15.23:0/1999739646 conn(0x7efdac0fb980 msgr2=0x7efdac1049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 shutdown_connections 2024-03-16T20:34:05.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.542+0000 7efdb36df640 1 -- 172.21.15.23:0/1999739646 wait complete. 2024-03-16T20:34:05.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 -- 172.21.15.23:0/3696041196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 msgr2=0x7fa38c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 --2- 172.21.15.23:0/3696041196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c100b60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009900 tx=0x7fa37c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 -- 172.21.15.23:0/3696041196 shutdown_connections 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 --2- 172.21.15.23:0/3696041196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 0x7fa38c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 --2- 172.21.15.23:0/3696041196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c100b60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 --2- 172.21.15.23:0/3696041196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 -- 172.21.15.23:0/3696041196 >> 172.21.15.23:0/3696041196 conn(0x7fa38c0fb130 msgr2=0x7fa38c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 -- 172.21.15.23:0/3696041196 shutdown_connections 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.655+0000 7fa392ea3640 1 -- 172.21.15.23:0/3696041196 wait complete. 2024-03-16T20:34:05.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 Processor -- start 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 -- start start 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 0x7fa38c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19f0c0 con 0x7fa38c1006e0 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19f230 con 0x7fa38c101a40 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa392ea3640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19f3f0 con 0x7fa38c1010a0 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49092/0 (socket says 172.21.15.23:49092) 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 -- 172.21.15.23:0/988827539 learned_addr learned my addr 172.21.15.23:0/988827539 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa383fff640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 0x7fa38c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa391440640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 -- 172.21.15.23:0/988827539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 msgr2=0x7fa38c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 0x7fa38c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 -- 172.21.15.23:0/988827539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 msgr2=0x7fa38c19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa390c3f640 1 -- 172.21.15.23:0/988827539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c009560 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.656+0000 7fa391440640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa390c3f640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c970 tx=0x7fa38800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388004220 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa388007e90 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c19f6d0 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38800f450 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.657+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c19fbb0 con 0x7fa38c1006e0 2024-03-16T20:34:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.658+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c068730 con 0x7fa38c1006e0 2024-03-16T20:34:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.659+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa388003610 con 0x7fa38c1006e0 2024-03-16T20:34:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.660+0000 7fa381ffb640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 0x7fa3740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.660+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa3880cd810 con 0x7fa38c1006e0 2024-03-16T20:34:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.660+0000 7fa383fff640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 0x7fa3740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.660+0000 7fa383fff640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 0x7fa3740a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009230 tx=0x7fa37c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.663+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3880967e0 con 0x7fa38c1006e0 2024-03-16T20:34:05.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3746022960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:05.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/189970634' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:05.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1999739646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:05.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.848+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa38c10c0b0 con 0x7fa38c1006e0 2024-03-16T20:34:05.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.849+0000 7fa381ffb640 1 -- 172.21.15.23:0/988827539 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fa388099ea0 con 0x7fa38c1006e0 2024-03-16T20:34:05.855 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:05.855 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:05.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.851+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 msgr2=0x7fa3740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.851+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 0x7fa3740a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009230 tx=0x7fa37c005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 msgr2=0x7fa38c1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c970 tx=0x7fa38800ce30 comp rx=0 tx=0).stop 2024-03-16T20:34:05.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 shutdown_connections 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1a30 0x7fa3740a3ef0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101a40 0x7fa38c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c1010a0 0x7fa38c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 --2- 172.21.15.23:0/988827539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c1006e0 0x7fa38c1975d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 >> 172.21.15.23:0/988827539 conn(0x7fa38c0fb130 msgr2=0x7fa38c0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 shutdown_connections 2024-03-16T20:34:05.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.852+0000 7fa392ea3640 1 -- 172.21.15.23:0/988827539 wait complete. 2024-03-16T20:34:05.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-16T20:34:05.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3746022960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:05.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/189970634' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:05.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1999739646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:05.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 -- 172.21.15.23:0/3198676498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 msgr2=0x7f23e8109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/3198676498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e8109620 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009ca0 tx=0x7f23d802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 -- 172.21.15.23:0/3198676498 shutdown_connections 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/3198676498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e8109620 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/3198676498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 0x7f23e8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/3198676498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e81043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 -- 172.21.15.23:0/3198676498 >> 172.21.15.23:0/3198676498 conn(0x7f23e80fb980 msgr2=0x7f23e80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 -- 172.21.15.23:0/3198676498 shutdown_connections 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.968+0000 7f23ed2f4640 1 -- 172.21.15.23:0/3198676498 wait complete. 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 Processor -- start 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 -- start start 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 0x7f23e8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e819f070 con 0x7f23e8107230 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e819f1e0 con 0x7f23e8101fd0 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23ed2f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e819f3a0 con 0x7f23e8104900 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49104/0 (socket says 172.21.15.23:49104) 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:55118/0 (socket says 172.21.15.23:55118) 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 -- 172.21.15.23:0/2915767213 learned_addr learned my addr 172.21.15.23:0/2915767213 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e67fc640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 0x7f23e8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 -- 172.21.15.23:0/2915767213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 msgr2=0x7f23e8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 0x7f23e8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.969+0000 7f23e77fe640 1 -- 172.21.15.23:0/2915767213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 msgr2=0x7f23e8197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23e77fe640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:05.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23e77fe640 1 -- 172.21.15.23:0/2915767213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8009900 con 0x7f23e8107230 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23e6ffd640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23e77fe640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23d8037040 tx=0x7f23d8009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80329b0 con 0x7f23e8107230 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e819f620 con 0x7f23e8107230 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23d8003e50 con 0x7f23e8107230 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80048a0 con 0x7f23e8107230 2024-03-16T20:34:05.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.970+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e819fad0 con 0x7f23e8107230 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.971+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e80fd470 con 0x7f23e8107230 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.972+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f23d800a040 con 0x7f23e8107230 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.973+0000 7f23dffff640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 0x7f23d40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.973+0000 7f23e6ffd640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 0x7f23d40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.973+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f23d80f07b0 con 0x7f23e8107230 2024-03-16T20:34:05.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.973+0000 7f23e6ffd640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 0x7f23d40a3ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009fc0 tx=0x7f23e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:05.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:05.976+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f23d80b9810 con 0x7f23e8107230 2024-03-16T20:34:06.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3746022960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:34:06.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/189970634' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:34:06.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1999739646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:06.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.197+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f23e810cf00 con 0x7f23e8107230 2024-03-16T20:34:06.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.692+0000 7f23dffff640 1 -- 172.21.15.23:0/2915767213 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f23d8034e60 con 0x7f23e8107230 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 msgr2=0x7f23d40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 0x7f23d40a3ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009fc0 tx=0x7f23e0005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 msgr2=0x7f23e819eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23d8037040 tx=0x7f23d8009360 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 shutdown_connections 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f23d40a1a30 0x7f23d40a3ef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e8107230 0x7f23e819eb30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e8104900 0x7f23e8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 --2- 172.21.15.23:0/2915767213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e8101fd0 0x7f23e8197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 >> 172.21.15.23:0/2915767213 conn(0x7f23e80fb980 msgr2=0x7f23e8102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 shutdown_connections 2024-03-16T20:34:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.694+0000 7f23ed2f4640 1 -- 172.21.15.23:0/2915767213 wait complete. 2024-03-16T20:34:06.712 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-16T20:34:06.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-16T20:34:06.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 -- 172.21.15.23:0/1192199719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c101f90 msgr2=0x7fac9c104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 --2- 172.21.15.23:0/1192199719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c101f90 0x7fac9c104380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009900 tx=0x7fac8c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 -- 172.21.15.23:0/1192199719 shutdown_connections 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 --2- 172.21.15.23:0/1192199719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1071f0 0x7fac9c109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 --2- 172.21.15.23:0/1192199719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c1048c0 0x7fac9c106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 --2- 172.21.15.23:0/1192199719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c101f90 0x7fac9c104380 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 -- 172.21.15.23:0/1192199719 >> 172.21.15.23:0/1192199719 conn(0x7fac9c0fb9a0 msgr2=0x7fac9c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 -- 172.21.15.23:0/1192199719 shutdown_connections 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.815+0000 7faca1e91640 1 -- 172.21.15.23:0/1192199719 wait complete. 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 Processor -- start 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 -- start start 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 0x7fac9c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1048c0 0x7fac9c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19f120 con 0x7fac9c1071f0 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19f290 con 0x7fac9c1048c0 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7faca1e91640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19f450 con 0x7fac9c101f90 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 0x7fac9c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49124/0 (socket says 172.21.15.23:49124) 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 -- 172.21.15.23:0/563051199 learned_addr learned my addr 172.21.15.23:0/563051199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9affd640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1048c0 0x7fac9c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 -- 172.21.15.23:0/563051199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 msgr2=0x7fac9c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 0x7fac9c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 -- 172.21.15.23:0/563051199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1048c0 msgr2=0x7fac9c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1048c0 0x7fac9c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:06.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9bfff640 1 -- 172.21.15.23:0/563051199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c009560 con 0x7fac9c1071f0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.816+0000 7fac9b7fe640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 0x7fac9c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7fac9bfff640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fac94009fd0 tx=0x7fac9400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9400bea0 con 0x7fac9c1071f0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac94009940 con 0x7fac9c1071f0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c19f730 con 0x7fac9c1071f0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac940106c0 con 0x7fac9c1071f0 2024-03-16T20:34:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.817+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c19fbf0 con 0x7fac9c1071f0 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.818+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c0fd490 con 0x7fac9c1071f0 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.819+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fac94004240 con 0x7fac9c1071f0 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.820+0000 7fac98ff9640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 0x7fac840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.820+0000 7fac9b7fe640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 0x7fac840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.820+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fac940cd1f0 con 0x7fac9c1071f0 2024-03-16T20:34:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.820+0000 7fac9b7fe640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 0x7fac840a3ef0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009230 tx=0x7fac8c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:06.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:06.823+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fac94096310 con 0x7fac9c1071f0 2024-03-16T20:34:06.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:06 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/988827539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:06.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:06 smithi189 ceph-mon[36655]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 924 B/s wr, 0 op/s 2024-03-16T20:34:06.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:06 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T20:34:07.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.045+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fac9c10cf20 con 0x7fac9c1071f0 2024-03-16T20:34:07.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:06 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/988827539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:07.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:06 smithi023 ceph-mon[31879]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 924 B/s wr, 0 op/s 2024-03-16T20:34:07.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:06 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T20:34:07.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:06 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/988827539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:07.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:06 smithi031 ceph-mon[36281]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 924 B/s wr, 0 op/s 2024-03-16T20:34:07.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:06 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-16T20:34:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.695+0000 7fac98ff9640 1 -- 172.21.15.23:0/563051199 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fac940999d0 con 0x7fac9c1071f0 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 msgr2=0x7fac840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 0x7fac840a3ef0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009230 tx=0x7fac8c005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 msgr2=0x7fac9c19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fac94009fd0 tx=0x7fac9400b260 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 shutdown_connections 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac840a1a30 0x7fac840a3ef0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac9c1071f0 0x7fac9c19ebe0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1048c0 0x7fac9c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 --2- 172.21.15.23:0/563051199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac9c101f90 0x7fac9c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.697+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 >> 172.21.15.23:0/563051199 conn(0x7fac9c0fb9a0 msgr2=0x7fac9c1028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.698+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 shutdown_connections 2024-03-16T20:34:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.698+0000 7faca1e91640 1 -- 172.21.15.23:0/563051199 wait complete. 2024-03-16T20:34:07.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/1671313917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b0101500 msgr2=0x7fe3b0101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/1671313917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b0101500 0x7fe3b0101920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009900 tx=0x7fe3a002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/1671313917 shutdown_connections 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/1671313917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01030b0 0x7fe3b0109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/1671313917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b01026f0 0x7fe3b0102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/1671313917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b0101500 0x7fe3b0101920 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.824+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/1671313917 >> 172.21.15.23:0/1671313917 conn(0x7fe3b00fca90 msgr2=0x7fe3b00feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/1671313917 shutdown_connections 2024-03-16T20:34:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/1671313917 wait complete. 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 Processor -- start 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- start start 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 0x7fe3b0197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01026f0 0x7fe3b0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b019f020 con 0x7fe3b01030b0 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b019f190 con 0x7fe3b0101500 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.825+0000 7fe3b5f8c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b019f350 con 0x7fe3b01026f0 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49128/0 (socket says 172.21.15.23:49128) 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 -- 172.21.15.23:0/355093960 learned_addr learned my addr 172.21.15.23:0/355093960 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3aeffd640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01026f0 0x7fe3b0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:07.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3af7fe640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 0x7fe3b0197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 -- 172.21.15.23:0/355093960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01026f0 msgr2=0x7fe3b0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01026f0 0x7fe3b0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 -- 172.21.15.23:0/355093960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 msgr2=0x7fe3b0197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 0x7fe3b0197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 -- 172.21.15.23:0/355093960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0009560 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3af7fe640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 0x7fe3b0197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3affff640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800eb10 tx=0x7fe3a800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a800ce20 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a8005020 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b019f630 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8010690 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.826+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b019fb40 con 0x7fe3b01030b0 2024-03-16T20:34:07.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.827+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b0108870 con 0x7fe3b01030b0 2024-03-16T20:34:07.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.829+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe3a8005500 con 0x7fe3b01030b0 2024-03-16T20:34:07.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.829+0000 7fe3acff9640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 0x7fe3980a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:07.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.830+0000 7fe3af7fe640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 0x7fe3980a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:07.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.830+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe3a8014070 con 0x7fe3b01030b0 2024-03-16T20:34:07.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.830+0000 7fe3af7fe640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 0x7fe3980a3ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0006010 tx=0x7fe3a0005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:07.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:07.833+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe3a8096e90 con 0x7fe3b01030b0 2024-03-16T20:34:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:07 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T20:34:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:07 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:07.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:07 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T20:34:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.056+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fe3b010cf10 con 0x7fe3b01030b0 2024-03-16T20:34:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:07 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T20:34:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:07 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:08.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:07 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T20:34:08.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:07 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2915767213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-16T20:34:08.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:07 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:08.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:07 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-16T20:34:08.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.707+0000 7fe3acff9640 1 -- 172.21.15.23:0/355093960 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fe3a8005190 con 0x7fe3b01030b0 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.710+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 msgr2=0x7fe3980a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.710+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 0x7fe3980a3ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0006010 tx=0x7fe3a0005be0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 msgr2=0x7fe3b019eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800eb10 tx=0x7fe3a800ef10 comp rx=0 tx=0).stop 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 shutdown_connections 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3980a1a30 0x7fe3980a3ef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b01030b0 0x7fe3b019eae0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b01026f0 0x7fe3b0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 --2- 172.21.15.23:0/355093960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b0101500 0x7fe3b0197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 >> 172.21.15.23:0/355093960 conn(0x7fe3b00fca90 msgr2=0x7fe3b0105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:08.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 shutdown_connections 2024-03-16T20:34:08.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.711+0000 7fe3b5f8c640 1 -- 172.21.15.23:0/355093960 wait complete. 2024-03-16T20:34:08.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 -- 172.21.15.23:0/2933050793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28075530 msgr2=0x7f6f28073980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 --2- 172.21.15.23:0/2933050793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28075530 0x7f6f28073980 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009930 tx=0x7f6f2402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 -- 172.21.15.23:0/2933050793 shutdown_connections 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 --2- 172.21.15.23:0/2933050793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f2810dae0 0x7f6f2810fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 --2- 172.21.15.23:0/2933050793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 0x7f6f280743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 --2- 172.21.15.23:0/2933050793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f28075530 0x7f6f28073980 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 -- 172.21.15.23:0/2933050793 >> 172.21.15.23:0/2933050793 conn(0x7f6f280fca80 msgr2=0x7f6f280feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 -- 172.21.15.23:0/2933050793 shutdown_connections 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.838+0000 7f6f30148640 1 -- 172.21.15.23:0/2933050793 wait complete. 2024-03-16T20:34:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 Processor -- start 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 -- start start 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 0x7f6f2819dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28075530 0x7f6f2819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a56c0 con 0x7f6f2810dae0 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a5830 con 0x7f6f28073f50 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f30148640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f281a59f0 con 0x7f6f28075530 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f2e6e5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f2e6e5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49146/0 (socket says 172.21.15.23:49146) 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f2e6e5640 1 -- 172.21.15.23:0/4135045865 learned_addr learned my addr 172.21.15.23:0/4135045865 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f2dee4640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 0x7f6f2819dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.839+0000 7f6f2d6e3640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28075530 0x7f6f2819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 -- 172.21.15.23:0/4135045865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28075530 msgr2=0x7f6f2819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28075530 0x7f6f2819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 -- 172.21.15.23:0/4135045865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 msgr2=0x7f6f2819dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 0x7f6f2819dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 -- 172.21.15.23:0/4135045865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24009560 con 0x7f6f2810dae0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f2e6e5640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000eb10 tx=0x7f6f2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004b20 con 0x7f6f2810dae0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f281a5cd0 con 0x7f6f2810dae0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f20004c80 con 0x7f6f2810dae0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20010670 con 0x7f6f2810dae0 2024-03-16T20:34:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.840+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281a61d0 con 0x7f6f2810dae0 2024-03-16T20:34:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.841+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f28103cb0 con 0x7f6f2810dae0 2024-03-16T20:34:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.842+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6f20010890 con 0x7f6f2810dae0 2024-03-16T20:34:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.843+0000 7f6f1effd640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.843+0000 7f6f2dee4640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:08.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.843+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6f20014070 con 0x7f6f2810dae0 2024-03-16T20:34:08.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.843+0000 7f6f2dee4640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009900 tx=0x7f6f2402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:08.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:08.846+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6f20096cf0 con 0x7f6f2810dae0 2024-03-16T20:34:08.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T20:34:08.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T20:34:08.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T20:34:08.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:08.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-16T20:34:08.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-16T20:34:08.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T20:34:08.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:08 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T20:34:09.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.072+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6f28113550 con 0x7f6f2810dae0 2024-03-16T20:34:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T20:34:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T20:34:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T20:34:09.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-16T20:34:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-16T20:34:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T20:34:09.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:08 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/563051199' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-16T20:34:09.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:08 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-16T20:34:09.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.925+0000 7f6f1effd640 1 -- 172.21.15.23:0/4135045865 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6f2009a3b0 con 0x7f6f2810dae0 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 msgr2=0x7f6f140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009900 tx=0x7f6f2402c7e0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 msgr2=0x7f6f281a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000eb10 tx=0x7f6f2000c2d0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 shutdown_connections 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6f140a1a30 0x7f6f140a3ef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2810dae0 0x7f6f281a5180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f28075530 0x7f6f2819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 --2- 172.21.15.23:0/4135045865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f28073f50 0x7f6f2819dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:09.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 >> 172.21.15.23:0/4135045865 conn(0x7f6f280fca80 msgr2=0x7f6f28100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:09.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.928+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 shutdown_connections 2024-03-16T20:34:09.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:09.929+0000 7f6f30148640 1 -- 172.21.15.23:0/4135045865 wait complete. 2024-03-16T20:34:09.951 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-16T20:34:09.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:active 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 9 up:standby 2024-03-16T20:34:09.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:active 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-16T20:34:09.972 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:09 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:active 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 9 up:standby 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:active 2024-03-16T20:34:09.978 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T20:34:09.979 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T20:34:09.979 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:09.979 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-16T20:34:09.979 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:09 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.058+0000 7fb0052fa640 1 -- 172.21.15.23:0/2867483801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 msgr2=0x7fb000100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.058+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2867483801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000100ba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009900 tx=0x7fafe802bcd0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 -- 172.21.15.23:0/2867483801 shutdown_connections 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2867483801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 0x7fb000101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2867483801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000100ba0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2867483801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 0x7fb000100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 -- 172.21.15.23:0/2867483801 >> 172.21.15.23:0/2867483801 conn(0x7fb0000fb980 msgr2=0x7fb0000fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 -- 172.21.15.23:0/2867483801 shutdown_connections 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 -- 172.21.15.23:0/2867483801 wait complete. 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 Processor -- start 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 -- start start 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.059+0000 7fb0052fa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 0x7fb0001997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 0x7fb0001a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001a12e0 con 0x7fb000100740 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001a1450 con 0x7fb0000ffde0 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0001a1610 con 0x7fb0001010e0 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49156/0 (socket says 172.21.15.23:49156) 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 -- 172.21.15.23:0/2264320260 learned_addr learned my addr 172.21.15.23:0/2264320260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffeffd640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 0x7fb0001997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fafff7fe640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 0x7fb0001a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 -- 172.21.15.23:0/2264320260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 msgr2=0x7fb0001a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 0x7fb0001a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 -- 172.21.15.23:0/2264320260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 msgr2=0x7fb0001997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 0x7fb0001997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 -- 172.21.15.23:0/2264320260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8009560 con 0x7fb000100740 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faffe7fc640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fafe8006010 tx=0x7fafe8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80043d0 con 0x7fb000100740 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafe80052d0 con 0x7fb000100740 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0001a1890 con 0x7fb000100740 2024-03-16T20:34:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8003cc0 con 0x7fb000100740 2024-03-16T20:34:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.060+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0001a1da0 con 0x7fb000100740 2024-03-16T20:34:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.061+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00010ad70 con 0x7fb000100740 2024-03-16T20:34:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.062+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fafe8005440 con 0x7fb000100740 2024-03-16T20:34:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.063+0000 7faff7fff640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 0x7fafec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.063+0000 7faffeffd640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 0x7fafec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.063+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fafe80effd0 con 0x7fb000100740 2024-03-16T20:34:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.063+0000 7faffeffd640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 0x7fafec0a3ef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faff8009bd0 tx=0x7faff8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.067+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fafe80b90c0 con 0x7fb000100740 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/355093960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] up:active 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:creating} 9 up:standby 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] up:active 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: daemon mds.l assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:creating} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-16T20:34:10.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:09 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-16T20:34:10.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.277+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb00002d000 con 0x7fafec0a1a30 2024-03-16T20:34:10.590 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:34:10.281+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:34:10.591 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:34:10.281+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:34:10.591 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:34:10.281+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:34:10.591 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:34:10.281+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:34:10.591 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:34:10.281+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:34:10.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.955+0000 7faff7fff640 1 -- 172.21.15.23:0/2264320260 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb00002d000 con 0x7fafec0a1a30 2024-03-16T20:34:10.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 msgr2=0x7fafec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 0x7fafec0a3ef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faff8009bd0 tx=0x7faff8005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 msgr2=0x7fb000199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fafe8006010 tx=0x7fafe8005e80 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 shutdown_connections 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fafec0a1a30 0x7fafec0a3ef0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0001010e0 0x7fb0001a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb000100740 0x7fb000199d30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 --2- 172.21.15.23:0/2264320260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0000ffde0 0x7fb0001997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 >> 172.21.15.23:0/2264320260 conn(0x7fb0000fb980 msgr2=0x7fb0000fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 shutdown_connections 2024-03-16T20:34:10.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:10.958+0000 7fb0052fa640 1 -- 172.21.15.23:0/2264320260 wait complete. 2024-03-16T20:34:10.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-16T20:34:11.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:active 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: pgmap v300: 129 pgs: 129 active+clean; 583 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:11.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:10 smithi023 ceph-mon[31879]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-16T20:34:11.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 -- 172.21.15.23:0/2558822842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70101f80 msgr2=0x7f3d70104370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/2558822842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70101f80 0x7f3d70104370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009900 tx=0x7f3d6002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 -- 172.21.15.23:0/2558822842 shutdown_connections 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/2558822842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701071e0 0x7f3d70109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/2558822842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d701048b0 0x7f3d70106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/2558822842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70101f80 0x7f3d70104370 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 -- 172.21.15.23:0/2558822842 >> 172.21.15.23:0/2558822842 conn(0x7f3d700fb990 msgr2=0x7f3d700fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 -- 172.21.15.23:0/2558822842 shutdown_connections 2024-03-16T20:34:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 -- 172.21.15.23:0/2558822842 wait complete. 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.121+0000 7f3d75e8f640 1 Processor -- start 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 -- start start 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d70101f80 0x7f3d70197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d701071e0 0x7f3d7019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019f170 con 0x7f3d701048b0 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019f2e0 con 0x7f3d701071e0 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019f4a0 con 0x7f3d70101f80 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59996/0 (socket says 172.21.15.23:59996) 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 -- 172.21.15.23:0/1460682625 learned_addr learned my addr 172.21.15.23:0/1460682625 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6ffff640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d701071e0 0x7f3d7019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6f7fe640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d70101f80 0x7f3d70197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 -- 172.21.15.23:0/1460682625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d70101f80 msgr2=0x7f3d70197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d70101f80 0x7f3d70197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 -- 172.21.15.23:0/1460682625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d701071e0 msgr2=0x7f3d7019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d701071e0 0x7f3d7019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 -- 172.21.15.23:0/1460682625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68009840 con 0x7f3d701048b0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6effd640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800e9f0 tx=0x7f3d5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580096c0 con 0x7f3d701048b0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d60009560 con 0x7f3d701048b0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d58005500 con 0x7f3d701048b0 2024-03-16T20:34:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.122+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58010430 con 0x7f3d701048b0 2024-03-16T20:34:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.123+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7019fab0 con 0x7f3d701048b0 2024-03-16T20:34:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.123+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d700fd480 con 0x7f3d701048b0 2024-03-16T20:34:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.124+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3d58005020 con 0x7f3d701048b0 2024-03-16T20:34:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.125+0000 7f3d6cff9640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 0x7f3d5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.125+0000 7f3d6f7fe640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 0x7f3d5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.125+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3d58014070 con 0x7f3d701048b0 2024-03-16T20:34:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.126+0000 7f3d6f7fe640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 0x7f3d5c0a3ef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009230 tx=0x7f3d60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.130+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3d58096930 con 0x7f3d701048b0 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:active 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: pgmap v300: 129 pgs: 129 active+clean; 583 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:11.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:10 smithi031 ceph-mon[36281]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-16T20:34:11.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:34:11.037+0000 7f8743f82700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-16T20:34:11.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:34:11.037+0000 7f8743f82700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-16T20:34:11.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:34:11.037+0000 7f8743f82700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-16T20:34:11.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:34:11.037+0000 7f8743f82700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4135045865' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] up:active 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=h=up:active,3=l=up:active} 3 up:standby-replay 5 up:standby 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: pgmap v300: 129 pgs: 129 active+clean; 583 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-16T20:34:11.222 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:10 smithi189 ceph-mon[36655]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-16T20:34:11.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.341+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d7002d000 con 0x7f3d5c0a1a30 2024-03-16T20:34:11.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.362+0000 7f3d6cff9640 1 -- 172.21.15.23:0/1460682625 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3d7002d000 con 0x7f3d5c0a1a30 2024-03-16T20:34:11.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 msgr2=0x7f3d5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 0x7f3d5c0a3ef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3d60009230 tx=0x7f3d60005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 msgr2=0x7f3d70197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800e9f0 tx=0x7f3d5800edf0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 shutdown_connections 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3d5c0a1a30 0x7f3d5c0a3ef0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d701071e0 0x7f3d7019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d701048b0 0x7f3d70197bc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 --2- 172.21.15.23:0/1460682625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d70101f80 0x7f3d70197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 >> 172.21.15.23:0/1460682625 conn(0x7f3d700fb990 msgr2=0x7f3d70102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 shutdown_connections 2024-03-16T20:34:11.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.364+0000 7f3d75e8f640 1 -- 172.21.15.23:0/1460682625 wait complete. 2024-03-16T20:34:11.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-16T20:34:11.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 -- 172.21.15.23:0/3800309039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8101fc0 msgr2=0x7fadf8108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 --2- 172.21.15.23:0/3800309039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8101fc0 0x7fadf8108850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009ca0 tx=0x7fadf002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 -- 172.21.15.23:0/3800309039 shutdown_connections 2024-03-16T20:34:11.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 --2- 172.21.15.23:0/3800309039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8101fc0 0x7fadf8108850 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 --2- 172.21.15.23:0/3800309039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 0x7fadf8101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 --2- 172.21.15.23:0/3800309039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8100410 0x7fadf8100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 -- 172.21.15.23:0/3800309039 >> 172.21.15.23:0/3800309039 conn(0x7fadf80fb9a0 msgr2=0x7fadf80fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 -- 172.21.15.23:0/3800309039 shutdown_connections 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.522+0000 7fadfd725640 1 -- 172.21.15.23:0/3800309039 wait complete. 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 Processor -- start 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 -- start start 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 0x7fadf8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8101fc0 0x7fadf819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819f060 con 0x7fadf8100410 2024-03-16T20:34:11.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819f1d0 con 0x7fadf8101fc0 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadfd725640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf819f390 con 0x7fadf8101600 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60002/0 (socket says 172.21.15.23:60002) 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 -- 172.21.15.23:0/1213446031 learned_addr learned my addr 172.21.15.23:0/1213446031 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf77fe640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8101fc0 0x7fadf819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadeffff640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 0x7fadf8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 -- 172.21.15.23:0/1213446031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 msgr2=0x7fadf8197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 0x7fadf8197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 -- 172.21.15.23:0/1213446031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8101fc0 msgr2=0x7fadf819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8101fc0 0x7fadf819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 -- 172.21.15.23:0/1213446031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8009490 con 0x7fadf8100410 2024-03-16T20:34:11.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.523+0000 7fadf6ffd640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fade800e9c0 tx=0x7fade800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.524+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade800cde0 con 0x7fadf8100410 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.524+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fade8010910 con 0x7fadf8100410 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.524+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf0009900 con 0x7fadf8100410 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.524+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8018a10 con 0x7fadf8100410 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.524+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf819f9f0 con 0x7fadf8100410 2024-03-16T20:34:11.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.525+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf810be20 con 0x7fadf8100410 2024-03-16T20:34:11.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.526+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fade8010a80 con 0x7fadf8100410 2024-03-16T20:34:11.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.526+0000 7fadf4ff9640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 0x7fade40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.526+0000 7fadeffff640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 0x7fade40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.526+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fade8014070 con 0x7fadf8100410 2024-03-16T20:34:11.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.527+0000 7fadeffff640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 0x7fade40a3ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fadf8198a80 tx=0x7fade0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.531+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fade8096ce0 con 0x7fadf8100410 2024-03-16T20:34:11.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.741+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fadf80fdb10 con 0x7fade40a1a30 2024-03-16T20:34:11.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.745+0000 7fadf4ff9640 1 -- 172.21.15.23:0/1213446031 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fadf80fdb10 con 0x7fade40a1a30 2024-03-16T20:34:11.751 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_0/2c33c834-1d14-4be6-b055-f67ccd042edc 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 msgr2=0x7fade40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 0x7fade40a3ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fadf8198a80 tx=0x7fade0005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 msgr2=0x7fadf8197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fade800e9c0 tx=0x7fade800ee80 comp rx=0 tx=0).stop 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 shutdown_connections 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fade40a1a30 0x7fade40a3ef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadf8101fc0 0x7fadf819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadf8101600 0x7fadf8197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 --2- 172.21.15.23:0/1213446031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadf8100410 0x7fadf8197570 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 >> 172.21.15.23:0/1213446031 conn(0x7fadf80fb9a0 msgr2=0x7fadf80fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:11.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.747+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 shutdown_connections 2024-03-16T20:34:11.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.748+0000 7fadfd725640 1 -- 172.21.15.23:0/1213446031 wait complete. 2024-03-16T20:34:11.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 -- 172.21.15.23:0/3397746753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 msgr2=0x7f1df0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 --2- 172.21.15.23:0/3397746753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df0102b80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009900 tx=0x7f1de402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 -- 172.21.15.23:0/3397746753 shutdown_connections 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 --2- 172.21.15.23:0/3397746753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df01030c0 0x7f1df0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 --2- 172.21.15.23:0/3397746753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df0102b80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 --2- 172.21.15.23:0/3397746753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df0101510 0x7f1df0101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 -- 172.21.15.23:0/3397746753 >> 172.21.15.23:0/3397746753 conn(0x7f1df00fca80 msgr2=0x7f1df00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 -- 172.21.15.23:0/3397746753 shutdown_connections 2024-03-16T20:34:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 -- 172.21.15.23:0/3397746753 wait complete. 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.865+0000 7f1df6148640 1 Processor -- start 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 -- start start 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 0x7f1df00752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 0x7f1df0073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df0074580 con 0x7f1df0102700 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df00746f0 con 0x7f1df0101510 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1df6148640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df01a1610 con 0x7f1df01030c0 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60018/0 (socket says 172.21.15.23:60018) 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 -- 172.21.15.23:0/3754780089 learned_addr learned my addr 172.21.15.23:0/3754780089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:11.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1def7fe640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 0x7f1df00752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deffff640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 0x7f1df0073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 -- 172.21.15.23:0/3754780089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 msgr2=0x7f1df0073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 0x7f1df0073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 -- 172.21.15.23:0/3754780089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 msgr2=0x7f1df00752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 0x7f1df00752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 -- 172.21.15.23:0/3754780089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4009560 con 0x7f1df0102700 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deffff640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 0x7f1df0073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1def7fe640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 0x7f1df00752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.866+0000 7f1deeffd640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1de4036040 tx=0x7f1de4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.867+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40043d0 con 0x7f1df0102700 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.867+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de40052d0 con 0x7f1df0102700 2024-03-16T20:34:11.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.867+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4003cc0 con 0x7f1df0102700 2024-03-16T20:34:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.867+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df01a1860 con 0x7f1df0102700 2024-03-16T20:34:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.867+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df01a1dc0 con 0x7f1df0102700 2024-03-16T20:34:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.868+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df0108830 con 0x7f1df0102700 2024-03-16T20:34:11.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.869+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1de4005440 con 0x7f1df0102700 2024-03-16T20:34:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.869+0000 7f1decff9640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 0x7f1dd80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.869+0000 7f1def7fe640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 0x7f1dd80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.869+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1de40efe40 con 0x7f1df0102700 2024-03-16T20:34:11.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.870+0000 7f1def7fe640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 0x7f1dd80a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1df0102560 tx=0x7f1de0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:11.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:11.873+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1de40b8e10 con 0x7f1df0102700 2024-03-16T20:34:12.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.083+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f1df002d000 con 0x7f1dd80a1ab0 2024-03-16T20:34:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:11 smithi023 ceph-mon[31879]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:11 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:active 2024-03-16T20:34:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:11 smithi023 ceph-mon[31879]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-16T20:34:12.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:11 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:34:12.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.107+0000 7f1decff9640 1 -- 172.21.15.23:0/3754780089 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1df002d000 con 0x7f1dd80a1ab0 2024-03-16T20:34:12.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 msgr2=0x7f1dd80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 0x7f1dd80a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1df0102560 tx=0x7f1de0005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:12.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 msgr2=0x7f1df00738e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1de4036040 tx=0x7f1de4005e80 comp rx=0 tx=0).stop 2024-03-16T20:34:12.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 shutdown_connections 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd80a1ab0 0x7f1dd80a3f70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1df01030c0 0x7f1df0073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1df0102700 0x7f1df00738e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 --2- 172.21.15.23:0/3754780089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df0101510 0x7f1df00752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 >> 172.21.15.23:0/3754780089 conn(0x7f1df00fca80 msgr2=0x7f1df0105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 shutdown_connections 2024-03-16T20:34:12.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.109+0000 7f1df6148640 1 -- 172.21.15.23:0/3754780089 wait complete. 2024-03-16T20:34:12.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-16T20:34:12.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-mon[36655]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:12.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:active 2024-03-16T20:34:12.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-mon[36655]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-16T20:34:12.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:11 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 -- 172.21.15.23:0/952817602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541003f0 msgr2=0x7f0b54100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/952817602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541003f0 0x7f0b54100810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009900 tx=0x7f0b4402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 -- 172.21.15.23:0/952817602 shutdown_connections 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/952817602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b54101fa0 0x7f0b54108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/952817602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b541015e0 0x7f0b54101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/952817602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541003f0 0x7f0b54100810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 -- 172.21.15.23:0/952817602 >> 172.21.15.23:0/952817602 conn(0x7f0b540fb980 msgr2=0x7f0b540fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 -- 172.21.15.23:0/952817602 shutdown_connections 2024-03-16T20:34:12.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.228+0000 7f0b5a56c640 1 -- 172.21.15.23:0/952817602 wait complete. 2024-03-16T20:34:12.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 Processor -- start 2024-03-16T20:34:12.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 -- start start 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 0x7f0b54197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b54101fa0 0x7f0b5419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5419f100 con 0x7f0b541015e0 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5419f270 con 0x7f0b541003f0 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b5a56c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5419f430 con 0x7f0b54101fa0 2024-03-16T20:34:12.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 0x7f0b54197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60034/0 (socket says 172.21.15.23:60034) 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b537fe640 1 -- 172.21.15.23:0/1232300046 learned_addr learned my addr 172.21.15.23:0/1232300046 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.229+0000 7f0b58b09640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b54101fa0 0x7f0b5419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 -- 172.21.15.23:0/1232300046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b54101fa0 msgr2=0x7f0b5419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b54101fa0 0x7f0b5419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 -- 172.21.15.23:0/1232300046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 msgr2=0x7f0b54197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 0x7f0b54197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 -- 172.21.15.23:0/1232300046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44009560 con 0x7f0b541015e0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b53fff640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 0x7f0b54197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b537fe640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800e9f0 tx=0x7f0b4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48009720 con 0x7f0b541015e0 2024-03-16T20:34:12.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b4800cd80 con 0x7f0b541015e0 2024-03-16T20:34:12.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5419f710 con 0x7f0b541015e0 2024-03-16T20:34:12.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48010690 con 0x7f0b541015e0 2024-03-16T20:34:12.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.230+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5419fbd0 con 0x7f0b541015e0 2024-03-16T20:34:12.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.231+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5410be00 con 0x7f0b541015e0 2024-03-16T20:34:12.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.232+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0b480046c0 con 0x7f0b541015e0 2024-03-16T20:34:12.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.233+0000 7f0b517fa640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 0x7f0b3c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.233+0000 7f0b53fff640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 0x7f0b3c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.233+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0b48014070 con 0x7f0b541015e0 2024-03-16T20:34:12.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.233+0000 7f0b53fff640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 0x7f0b3c0a3ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b44005ab0 tx=0x7f0b44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.237+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0b48096980 con 0x7f0b541015e0 2024-03-16T20:34:12.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:11 smithi031 ceph-mon[36281]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:12.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:11 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] up:active 2024-03-16T20:34:12.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:11 smithi031 ceph-mon[36281]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-16T20:34:12.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:11 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:34:12.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.462+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b54068d90 con 0x7f0b3c0a1a30 2024-03-16T20:34:12.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.465+0000 7f0b517fa640 1 -- 172.21.15.23:0/1232300046 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f0b54068d90 con 0x7f0b3c0a1a30 2024-03-16T20:34:12.470 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_1/5fa01b57-dcf5-4fa0-963f-99b4c0e05eb9 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.467+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 msgr2=0x7f0b3c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.467+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 0x7f0b3c0a3ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b44005ab0 tx=0x7f0b44005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.467+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 msgr2=0x7f0b54197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.467+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800e9f0 tx=0x7f0b4800edf0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 shutdown_connections 2024-03-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b3c0a1a30 0x7f0b3c0a3ef0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b54101fa0 0x7f0b5419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b541015e0 0x7f0b54197b50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 --2- 172.21.15.23:0/1232300046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b541003f0 0x7f0b54197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 >> 172.21.15.23:0/1232300046 conn(0x7f0b540fb980 msgr2=0x7f0b54068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 shutdown_connections 2024-03-16T20:34:12.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.468+0000 7f0b5a56c640 1 -- 172.21.15.23:0/1232300046 wait complete. 2024-03-16T20:34:12.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- 172.21.15.23:0/2613424631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24101770 msgr2=0x7fec24108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 --2- 172.21.15.23:0/2613424631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24101770 0x7fec24108000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009be0 tx=0x7fec1c02bef0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- 172.21.15.23:0/2613424631 shutdown_connections 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 --2- 172.21.15.23:0/2613424631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24101770 0x7fec24108000 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 --2- 172.21.15.23:0/2613424631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec24100db0 0x7fec24101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 --2- 172.21.15.23:0/2613424631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec240ffbc0 0x7fec240fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- 172.21.15.23:0/2613424631 >> 172.21.15.23:0/2613424631 conn(0x7fec240fb130 msgr2=0x7fec240fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- 172.21.15.23:0/2613424631 shutdown_connections 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- 172.21.15.23:0/2613424631 wait complete. 2024-03-16T20:34:12.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 Processor -- start 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.582+0000 7fec29f16640 1 -- start start 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24100db0 0x7fec241969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec24101770 0x7fec2419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2419df50 con 0x7fec240ffbc0 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2419e0c0 con 0x7fec24100db0 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2419e280 con 0x7fec24101770 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60052/0 (socket says 172.21.15.23:60052) 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 -- 172.21.15.23:0/107274482 learned_addr learned my addr 172.21.15.23:0/107274482 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec22ffd640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24100db0 0x7fec241969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec23fff640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec24101770 0x7fec2419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 -- 172.21.15.23:0/107274482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec24101770 msgr2=0x7fec2419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec24101770 0x7fec2419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 -- 172.21.15.23:0/107274482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24100db0 msgr2=0x7fec241969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24100db0 0x7fec241969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 -- 172.21.15.23:0/107274482 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c009560 con 0x7fec240ffbc0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec237fe640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fec1400d950 tx=0x7fec1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14009c30 con 0x7fec240ffbc0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c009840 con 0x7fec240ffbc0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec14009d90 con 0x7fec240ffbc0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1400f720 con 0x7fec240ffbc0 2024-03-16T20:34:12.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.583+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2419e8e0 con 0x7fec240ffbc0 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.584+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec24068730 con 0x7fec240ffbc0 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.585+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fec14010a60 con 0x7fec240ffbc0 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.586+0000 7fec20ff9640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 0x7fec100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.586+0000 7fec22ffd640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 0x7fec100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.586+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fec140cd290 con 0x7fec240ffbc0 2024-03-16T20:34:12.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.587+0000 7fec22ffd640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 0x7fec100a3ef0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec24197970 tx=0x7fec0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.590+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fec14096330 con 0x7fec240ffbc0 2024-03-16T20:34:12.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.810+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec2410b5c0 con 0x7fec240ffbc0 2024-03-16T20:34:12.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.811+0000 7fec20ff9640 1 -- 172.21.15.23:0/107274482 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7fec14021090 con 0x7fec240ffbc0 2024-03-16T20:34:12.816 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:12.816 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:34:12.817 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 msgr2=0x7fec100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 0x7fec100a3ef0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec24197970 tx=0x7fec0c009260 comp rx=0 tx=0).stop 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 msgr2=0x7fec241964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fec1400d950 tx=0x7fec1400de10 comp rx=0 tx=0).stop 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 shutdown_connections 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fec100a1a30 0x7fec100a3ef0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec24101770 0x7fec2419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec24100db0 0x7fec241969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 --2- 172.21.15.23:0/107274482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec240ffbc0 0x7fec241964b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 >> 172.21.15.23:0/107274482 conn(0x7fec240fb130 msgr2=0x7fec24104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 shutdown_connections 2024-03-16T20:34:12.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.813+0000 7fec29f16640 1 -- 172.21.15.23:0/107274482 wait complete. 2024-03-16T20:34:12.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 -- 172.21.15.23:0/523168564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd780ffde0 msgr2=0x7fcd78100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 --2- 172.21.15.23:0/523168564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd780ffde0 0x7fcd78100200 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009900 tx=0x7fcd7002bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 -- 172.21.15.23:0/523168564 shutdown_connections 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 --2- 172.21.15.23:0/523168564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd781010e0 0x7fcd78101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 --2- 172.21.15.23:0/523168564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd78100740 0x7fcd78100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 --2- 172.21.15.23:0/523168564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd780ffde0 0x7fcd78100200 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 -- 172.21.15.23:0/523168564 >> 172.21.15.23:0/523168564 conn(0x7fcd780fb980 msgr2=0x7fcd780fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 -- 172.21.15.23:0/523168564 shutdown_connections 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.927+0000 7fcd7c981640 1 -- 172.21.15.23:0/523168564 wait complete. 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 Processor -- start 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 -- start start 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd780ffde0 0x7fcd781997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd781010e0 0x7fcd781a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd781a12b0 con 0x7fcd78100740 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd781a1420 con 0x7fcd781010e0 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd7c981640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd781a15e0 con 0x7fcd780ffde0 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd75d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60068/0 (socket says 172.21.15.23:60068) 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.928+0000 7fcd75d9b640 1 -- 172.21.15.23:0/1830348029 learned_addr learned my addr 172.21.15.23:0/1830348029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd7659c640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd780ffde0 0x7fcd781997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd76d9d640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd781010e0 0x7fcd781a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 -- 172.21.15.23:0/1830348029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd780ffde0 msgr2=0x7fcd781997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd780ffde0 0x7fcd781997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 -- 172.21.15.23:0/1830348029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd781010e0 msgr2=0x7fcd781a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd781010e0 0x7fcd781a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 -- 172.21.15.23:0/1830348029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd70009560 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd75d9b640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00e9f0 tx=0x7fcd6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c009700 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd781a18c0 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd6c00cd80 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c018920 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.929+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd781a1e20 con 0x7fcd78100740 2024-03-16T20:34:12.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.930+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7810acb0 con 0x7fcd78100740 2024-03-16T20:34:12.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.931+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcd6c018a80 con 0x7fcd78100740 2024-03-16T20:34:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.932+0000 7fcd677fe640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 0x7fcd600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.932+0000 7fcd7659c640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 0x7fcd600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.932+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcd6c014070 con 0x7fcd78100740 2024-03-16T20:34:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.932+0000 7fcd7659c640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 0x7fcd600a3ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009230 tx=0x7fcd70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.936+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcd6c053280 con 0x7fcd78100740 2024-03-16T20:34:12.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:12.967+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fcd6c096e00 con 0x7fcd78100740 2024-03-16T20:34:13.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:12 smithi023 ceph-mon[31879]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:12 smithi023 ceph-mon[31879]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:12 smithi023 ceph-mon[31879]: pgmap v301: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T20:34:13.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:12 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/107274482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:34:13.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.125+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd7802cf60 con 0x7fcd78100740 2024-03-16T20:34:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.127+0000 7fcd677fe640 1 -- 172.21.15.23:0/1830348029 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fcd6c02b900 con 0x7fcd78100740 2024-03-16T20:34:13.132 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:13.132 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 msgr2=0x7fcd600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 0x7fcd600a3ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009230 tx=0x7fcd70005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 msgr2=0x7fcd78199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00e9f0 tx=0x7fcd6c00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 shutdown_connections 2024-03-16T20:34:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcd600a1a30 0x7fcd600a3ef0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd781010e0 0x7fcd781a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd78100740 0x7fcd78199d00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 --2- 172.21.15.23:0/1830348029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd780ffde0 0x7fcd781997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 >> 172.21.15.23:0/1830348029 conn(0x7fcd780fb980 msgr2=0x7fcd780fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 shutdown_connections 2024-03-16T20:34:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.129+0000 7fcd7c981640 1 -- 172.21.15.23:0/1830348029 wait complete. 2024-03-16T20:34:13.145 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-16T20:34:13.146 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:34:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- 172.21.15.23:0/2851295141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101fc0 msgr2=0x7f7188108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 --2- 172.21.15.23:0/2851295141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101fc0 0x7f7188108850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7184009ca0 tx=0x7f718402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- 172.21.15.23:0/2851295141 shutdown_connections 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 --2- 172.21.15.23:0/2851295141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101fc0 0x7f7188108850 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 --2- 172.21.15.23:0/2851295141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101600 0x7f7188101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 --2- 172.21.15.23:0/2851295141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f7188100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- 172.21.15.23:0/2851295141 >> 172.21.15.23:0/2851295141 conn(0x7f71880fb980 msgr2=0x7f71880fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- 172.21.15.23:0/2851295141 shutdown_connections 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- 172.21.15.23:0/2851295141 wait complete. 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 Processor -- start 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.243+0000 7f71900f4640 1 -- start start 2024-03-16T20:34:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f71881975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101fc0 0x7f718819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718819f090 con 0x7f7188101600 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718819f200 con 0x7f7188100410 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718819f3c0 con 0x7f7188101fc0 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60098/0 (socket says 172.21.15.23:60098) 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 -- 172.21.15.23:0/3991249096 learned_addr learned my addr 172.21.15.23:0/3991249096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718de90640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f71881975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718e691640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101fc0 0x7f718819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 -- 172.21.15.23:0/3991249096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101fc0 msgr2=0x7f718819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101fc0 0x7f718819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 -- 172.21.15.23:0/3991249096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 msgr2=0x7f71881975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f71881975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 -- 172.21.15.23:0/3991249096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184009900 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718de90640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f71881975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f718d68f640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f717800e9f0 tx=0x7f717800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178009720 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f717800cd80 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718819f6a0 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178010690 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.244+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718819fb60 con 0x7f7188101600 2024-03-16T20:34:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.245+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718810be20 con 0x7f7188101600 2024-03-16T20:34:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.246+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f71780046c0 con 0x7f7188101600 2024-03-16T20:34:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.247+0000 7f717effd640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 0x7f71740a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.247+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7178014070 con 0x7f7188101600 2024-03-16T20:34:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.247+0000 7f718de90640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 0x7f71740a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.247+0000 7f718de90640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 0x7f71740a3f80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7188101460 tx=0x7f7180005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.252+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f71780969b0 con 0x7f7188101600 2024-03-16T20:34:13.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.429+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71880fdb10 con 0x7f71740a1ac0 2024-03-16T20:34:13.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:12 smithi031 ceph-mon[36281]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:12 smithi031 ceph-mon[36281]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:12 smithi031 ceph-mon[36281]: pgmap v301: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T20:34:13.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:12 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/107274482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:34:13.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.439+0000 7f717effd640 1 -- 172.21.15.23:0/3991249096 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340609 (secure 0 0 0) 0x7f71880fdb10 con 0x7f71740a1ac0 2024-03-16T20:34:13.448 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 msgr2=0x7f71740a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 0x7f71740a3f80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7188101460 tx=0x7f7180005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 msgr2=0x7f7188197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f717800e9f0 tx=0x7f717800edf0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 shutdown_connections 2024-03-16T20:34:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71740a1ac0 0x7f71740a3f80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7188101fc0 0x7f718819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7188101600 0x7f7188197ae0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 --2- 172.21.15.23:0/3991249096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7188100410 0x7f71881975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 >> 172.21.15.23:0/3991249096 conn(0x7f71880fb980 msgr2=0x7f71880fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 shutdown_connections 2024-03-16T20:34:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.442+0000 7f71900f4640 1 -- 172.21.15.23:0/3991249096 wait complete. 2024-03-16T20:34:13.459 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-16T20:34:11.944681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619343,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":163,"num_read_kb":143,"num_write":388,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373496,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568328,"statfs":{"total":1151940427776,"available":1151557967872,"internally_reserved":0,"allocated":12140544,"data_stored":7597567,"data_compressed":285306,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26565,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":37,"num_read_kb":34,"num_write":162,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156791+0000","last_change":"2024-03-16T20:33:49.858111+0000","last_active":"2024-03-16T20:33:55.156791+0000","last_peered":"2024-03-16T20:33:55.156791+0000","last_clean":"2024-03-16T20:33:55.156791+0000","last_became_active":"2024-03-16T20:33:49.857656+0000","last_became_peered":"2024-03-16T20:33:49.857656+0000","last_unstale":"2024-03-16T20:33:55.156791+0000","last_undegraded":"2024-03-16T20:33:55.156791+0000","last_fullsized":"2024-03-16T20:33:55.156791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:16:47.779595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110728+0000","last_change":"2024-03-16T20:33:49.854850+0000","last_active":"2024-03-16T20:33:55.110728+0000","last_peered":"2024-03-16T20:33:55.110728+0000","last_clean":"2024-03-16T20:33:55.110728+0000","last_became_active":"2024-03-16T20:33:49.854399+0000","last_became_peered":"2024-03-16T20:33:49.854399+0000","last_unstale":"2024-03-16T20:33:55.110728+0000","last_undegraded":"2024-03-16T20:33:55.110728+0000","last_fullsized":"2024-03-16T20:33:55.110728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:55.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111050+0000","last_change":"2024-03-16T20:33:49.858831+0000","last_active":"2024-03-16T20:33:55.111050+0000","last_peered":"2024-03-16T20:33:55.111050+0000","last_clean":"2024-03-16T20:33:55.111050+0000","last_became_active":"2024-03-16T20:33:49.858507+0000","last_became_peered":"2024-03-16T20:33:49.858507+0000","last_unstale":"2024-03-16T20:33:55.111050+0000","last_undegraded":"2024-03-16T20:33:55.111050+0000","last_fullsized":"2024-03-16T20:33:55.111050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:35.447285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111877+0000","last_change":"2024-03-16T20:33:49.852269+0000","last_active":"2024-03-16T20:33:55.111877+0000","last_peered":"2024-03-16T20:33:55.111877+0000","last_clean":"2024-03-16T20:33:55.111877+0000","last_became_active":"2024-03-16T20:33:49.851904+0000","last_became_peered":"2024-03-16T20:33:49.851904+0000","last_unstale":"2024-03-16T20:33:55.111877+0000","last_undegraded":"2024-03-16T20:33:55.111877+0000","last_fullsized":"2024-03-16T20:33:55.111877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:38.520111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109628+0000","last_change":"2024-03-16T20:33:49.852371+0000","last_active":"2024-03-16T20:33:55.109628+0000","last_peered":"2024-03-16T20:33:55.109628+0000","last_clean":"2024-03-16T20:33:55.109628+0000","last_became_active":"2024-03-16T20:33:49.852034+0000","last_became_peered":"2024-03-16T20:33:49.852034+0000","last_unstale":"2024-03-16T20:33:55.109628+0000","last_undegraded":"2024-03-16T20:33:55.109628+0000","last_fullsized":"2024-03-16T20:33:55.109628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:10.229852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110839+0000","last_change":"2024-03-16T20:33:49.855209+0000","last_active":"2024-03-16T20:33:55.110839+0000","last_peered":"2024-03-16T20:33:55.110839+0000","last_clean":"2024-03-16T20:33:55.110839+0000","last_became_active":"2024-03-16T20:33:49.854502+0000","last_became_peered":"2024-03-16T20:33:49.854502+0000","last_unstale":"2024-03-16T20:33:55.110839+0000","last_undegraded":"2024-03-16T20:33:55.110839+0000","last_fullsized":"2024-03-16T20:33:55.110839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:47.792663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110377+0000","last_change":"2024-03-16T20:33:49.852668+0000","last_active":"2024-03-16T20:33:55.110377+0000","last_peered":"2024-03-16T20:33:55.110377+0000","last_clean":"2024-03-16T20:33:55.110377+0000","last_became_active":"2024-03-16T20:33:49.852263+0000","last_became_peered":"2024-03-16T20:33:49.852263+0000","last_unstale":"2024-03-16T20:33:55.110377+0000","last_undegraded":"2024-03-16T20:33:55.110377+0000","last_fullsized":"2024-03-16T20:33:55.110377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:49.815894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110713+0000","last_change":"2024-03-16T20:33:49.854962+0000","last_active":"2024-03-16T20:33:55.110713+0000","last_peered":"2024-03-16T20:33:55.110713+0000","last_clean":"2024-03-16T20:33:55.110713+0000","last_became_active":"2024-03-16T20:33:49.853963+0000","last_became_peered":"2024-03-16T20:33:49.853963+0000","last_unstale":"2024-03-16T20:33:55.110713+0000","last_undegraded":"2024-03-16T20:33:55.110713+0000","last_fullsized":"2024-03-16T20:33:55.110713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:37:19.213607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871953+0000","last_change":"2024-03-16T20:33:49.857741+0000","last_active":"2024-03-16T20:33:54.871953+0000","last_peered":"2024-03-16T20:33:54.871953+0000","last_clean":"2024-03-16T20:33:54.871953+0000","last_became_active":"2024-03-16T20:33:49.857354+0000","last_became_peered":"2024-03-16T20:33:49.857354+0000","last_unstale":"2024-03-16T20:33:54.871953+0000","last_undegraded":"2024-03-16T20:33:54.871953+0000","last_fullsized":"2024-03-16T20:33:54.871953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:22.085460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.112211+0000","last_change":"2024-03-16T20:33:49.864008+0000","last_active":"2024-03-16T20:33:55.112211+0000","last_peered":"2024-03-16T20:33:55.112211+0000","last_clean":"2024-03-16T20:33:55.112211+0000","last_became_active":"2024-03-16T20:33:49.862779+0000","last_became_peered":"2024-03-16T20:33:49.862779+0000","last_unstale":"2024-03-16T20:33:55.112211+0000","last_undegraded":"2024-03-16T20:33:55.112211+0000","last_fullsized":"2024-03-16T20:33:55.112211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:34:23.249973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110994+0000","last_change":"2024-03-16T20:33:49.858951+0000","last_active":"2024-03-16T20:33:55.110994+0000","last_peered":"2024-03-16T20:33:55.110994+0000","last_clean":"2024-03-16T20:33:55.110994+0000","last_became_active":"2024-03-16T20:33:49.858575+0000","last_became_peered":"2024-03-16T20:33:49.858575+0000","last_unstale":"2024-03-16T20:33:55.110994+0000","last_undegraded":"2024-03-16T20:33:55.110994+0000","last_fullsized":"2024-03-16T20:33:55.110994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:07:01.030826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111765+0000","last_change":"2024-03-16T20:33:49.854399+0000","last_active":"2024-03-16T20:33:55.111765+0000","last_peered":"2024-03-16T20:33:55.111765+0000","last_clean":"2024-03-16T20:33:55.111765+0000","last_became_active":"2024-03-16T20:33:49.853937+0000","last_became_peered":"2024-03-16T20:33:49.853937+0000","last_unstale":"2024-03-16T20:33:55.111765+0000","last_undegraded":"2024-03-16T20:33:55.111765+0000","last_fullsized":"2024-03-16T20:33:55.111765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:48:00.207213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873323+0000","last_change":"2024-03-16T20:33:49.853101+0000","last_active":"2024-03-16T20:33:54.873323+0000","last_peered":"2024-03-16T20:33:54.873323+0000","last_clean":"2024-03-16T20:33:54.873323+0000","last_became_active":"2024-03-16T20:33:49.852667+0000","last_became_peered":"2024-03-16T20:33:49.852667+0000","last_unstale":"2024-03-16T20:33:54.873323+0000","last_undegraded":"2024-03-16T20:33:54.873323+0000","last_fullsized":"2024-03-16T20:33:54.873323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:17:43.047348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110331+0000","last_change":"2024-03-16T20:33:49.852907+0000","last_active":"2024-03-16T20:33:55.110331+0000","last_peered":"2024-03-16T20:33:55.110331+0000","last_clean":"2024-03-16T20:33:55.110331+0000","last_became_active":"2024-03-16T20:33:49.852552+0000","last_became_peered":"2024-03-16T20:33:49.852552+0000","last_unstale":"2024-03-16T20:33:55.110331+0000","last_undegraded":"2024-03-16T20:33:55.110331+0000","last_fullsized":"2024-03-16T20:33:55.110331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:46:34.019736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110107+0000","last_change":"2024-03-16T20:33:49.863477+0000","last_active":"2024-03-16T20:33:55.110107+0000","last_peered":"2024-03-16T20:33:55.110107+0000","last_clean":"2024-03-16T20:33:55.110107+0000","last_became_active":"2024-03-16T20:33:49.863163+0000","last_became_peered":"2024-03-16T20:33:49.863163+0000","last_unstale":"2024-03-16T20:33:55.110107+0000","last_undegraded":"2024-03-16T20:33:55.110107+0000","last_fullsized":"2024-03-16T20:33:55.110107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:46:55.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.870977+0000","last_change":"2024-03-16T20:33:49.863312+0000","last_active":"2024-03-16T20:33:54.870977+0000","last_peered":"2024-03-16T20:33:54.870977+0000","last_clean":"2024-03-16T20:33:54.870977+0000","last_became_active":"2024-03-16T20:33:49.862918+0000","last_became_peered":"2024-03-16T20:33:49.862918+0000","last_unstale":"2024-03-16T20:33:54.870977+0000","last_undegraded":"2024-03-16T20:33:54.870977+0000","last_fullsized":"2024-03-16T20:33:54.870977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:33.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156624+0000","last_change":"2024-03-16T20:33:49.858863+0000","last_active":"2024-03-16T20:33:55.156624+0000","last_peered":"2024-03-16T20:33:55.156624+0000","last_clean":"2024-03-16T20:33:55.156624+0000","last_became_active":"2024-03-16T20:33:49.858543+0000","last_became_peered":"2024-03-16T20:33:49.858543+0000","last_unstale":"2024-03-16T20:33:55.156624+0000","last_undegraded":"2024-03-16T20:33:55.156624+0000","last_fullsized":"2024-03-16T20:33:55.156624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:04.584473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110362+0000","last_change":"2024-03-16T20:33:49.857863+0000","last_active":"2024-03-16T20:33:55.110362+0000","last_peered":"2024-03-16T20:33:55.110362+0000","last_clean":"2024-03-16T20:33:55.110362+0000","last_became_active":"2024-03-16T20:33:49.857503+0000","last_became_peered":"2024-03-16T20:33:49.857503+0000","last_unstale":"2024-03-16T20:33:55.110362+0000","last_undegraded":"2024-03-16T20:33:55.110362+0000","last_fullsized":"2024-03-16T20:33:55.110362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:11.206926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110192+0000","last_change":"2024-03-16T20:33:49.849940+0000","last_active":"2024-03-16T20:33:55.110192+0000","last_peered":"2024-03-16T20:33:55.110192+0000","last_clean":"2024-03-16T20:33:55.110192+0000","last_became_active":"2024-03-16T20:33:49.849632+0000","last_became_peered":"2024-03-16T20:33:49.849632+0000","last_unstale":"2024-03-16T20:33:55.110192+0000","last_undegraded":"2024-03-16T20:33:55.110192+0000","last_fullsized":"2024-03-16T20:33:55.110192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:25.921270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109900+0000","last_change":"2024-03-16T20:33:53.882456+0000","last_active":"2024-03-16T20:33:55.109900+0000","last_peered":"2024-03-16T20:33:55.109900+0000","last_clean":"2024-03-16T20:33:55.109900+0000","last_became_active":"2024-03-16T20:33:53.882040+0000","last_became_peered":"2024-03-16T20:33:53.882040+0000","last_unstale":"2024-03-16T20:33:55.109900+0000","last_undegraded":"2024-03-16T20:33:55.109900+0000","last_fullsized":"2024-03-16T20:33:55.109900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:59.392600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.942164+0000","last_change":"2024-03-16T20:33:48.846129+0000","last_active":"2024-03-16T20:34:09.942164+0000","last_peered":"2024-03-16T20:34:09.942164+0000","last_clean":"2024-03-16T20:34:09.942164+0000","last_became_active":"2024-03-16T20:33:48.845783+0000","last_became_peered":"2024-03-16T20:33:48.845783+0000","last_unstale":"2024-03-16T20:34:09.942164+0000","last_undegraded":"2024-03-16T20:34:09.942164+0000","last_fullsized":"2024-03-16T20:34:09.942164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:46:03.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110676+0000","last_change":"2024-03-16T20:33:49.851397+0000","last_active":"2024-03-16T20:33:55.110676+0000","last_peered":"2024-03-16T20:33:55.110676+0000","last_clean":"2024-03-16T20:33:55.110676+0000","last_became_active":"2024-03-16T20:33:49.851097+0000","last_became_peered":"2024-03-16T20:33:49.851097+0000","last_unstale":"2024-03-16T20:33:55.110676+0000","last_undegraded":"2024-03-16T20:33:55.110676+0000","last_fullsized":"2024-03-16T20:33:55.110676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:57:55.165017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111379+0000","last_change":"2024-03-16T20:33:53.893593+0000","last_active":"2024-03-16T20:33:55.111379+0000","last_peered":"2024-03-16T20:33:55.111379+0000","last_clean":"2024-03-16T20:33:55.111379+0000","last_became_active":"2024-03-16T20:33:53.893101+0000","last_became_peered":"2024-03-16T20:33:53.893101+0000","last_unstale":"2024-03-16T20:33:55.111379+0000","last_undegraded":"2024-03-16T20:33:55.111379+0000","last_fullsized":"2024-03-16T20:33:55.111379+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:22:59.472049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111001+0000","last_change":"2024-03-16T20:33:48.852786+0000","last_active":"2024-03-16T20:33:55.111001+0000","last_peered":"2024-03-16T20:33:55.111001+0000","last_clean":"2024-03-16T20:33:55.111001+0000","last_became_active":"2024-03-16T20:33:48.852477+0000","last_became_peered":"2024-03-16T20:33:48.852477+0000","last_unstale":"2024-03-16T20:33:55.111001+0000","last_undegraded":"2024-03-16T20:33:55.111001+0000","last_fullsized":"2024-03-16T20:33:55.111001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:49.191312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111377+0000","last_change":"2024-03-16T20:33:49.859123+0000","last_active":"2024-03-16T20:33:55.111377+0000","last_peered":"2024-03-16T20:33:55.111377+0000","last_clean":"2024-03-16T20:33:55.111377+0000","last_became_active":"2024-03-16T20:33:49.858746+0000","last_became_peered":"2024-03-16T20:33:49.858746+0000","last_unstale":"2024-03-16T20:33:55.111377+0000","last_undegraded":"2024-03-16T20:33:55.111377+0000","last_fullsized":"2024-03-16T20:33:55.111377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:35.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111721+0000","last_change":"2024-03-16T20:33:53.891821+0000","last_active":"2024-03-16T20:33:55.111721+0000","last_peered":"2024-03-16T20:33:55.111721+0000","last_clean":"2024-03-16T20:33:55.111721+0000","last_became_active":"2024-03-16T20:33:53.891383+0000","last_became_peered":"2024-03-16T20:33:53.891383+0000","last_unstale":"2024-03-16T20:33:55.111721+0000","last_undegraded":"2024-03-16T20:33:55.111721+0000","last_fullsized":"2024-03-16T20:33:55.111721+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:39:03.784185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.942046+0000","last_change":"2024-03-16T20:33:48.845405+0000","last_active":"2024-03-16T20:34:09.942046+0000","last_peered":"2024-03-16T20:34:09.942046+0000","last_clean":"2024-03-16T20:34:09.942046+0000","last_became_active":"2024-03-16T20:33:48.845033+0000","last_became_peered":"2024-03-16T20:33:48.845033+0000","last_unstale":"2024-03-16T20:34:09.942046+0000","last_undegraded":"2024-03-16T20:34:09.942046+0000","last_fullsized":"2024-03-16T20:34:09.942046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:52.632960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871503+0000","last_change":"2024-03-16T20:33:49.864934+0000","last_active":"2024-03-16T20:33:54.871503+0000","last_peered":"2024-03-16T20:33:54.871503+0000","last_clean":"2024-03-16T20:33:54.871503+0000","last_became_active":"2024-03-16T20:33:49.864574+0000","last_became_peered":"2024-03-16T20:33:49.864574+0000","last_unstale":"2024-03-16T20:33:54.871503+0000","last_undegraded":"2024-03-16T20:33:54.871503+0000","last_fullsized":"2024-03-16T20:33:54.871503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:07.010788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111913+0000","last_change":"2024-03-16T20:33:53.890446+0000","last_active":"2024-03-16T20:33:55.111913+0000","last_peered":"2024-03-16T20:33:55.111913+0000","last_clean":"2024-03-16T20:33:55.111913+0000","last_became_active":"2024-03-16T20:33:53.889832+0000","last_became_peered":"2024-03-16T20:33:53.889832+0000","last_unstale":"2024-03-16T20:33:55.111913+0000","last_undegraded":"2024-03-16T20:33:55.111913+0000","last_fullsized":"2024-03-16T20:33:55.111913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:49.031999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940746+0000","last_change":"2024-03-16T20:33:48.845809+0000","last_active":"2024-03-16T20:34:09.940746+0000","last_peered":"2024-03-16T20:34:09.940746+0000","last_clean":"2024-03-16T20:34:09.940746+0000","last_became_active":"2024-03-16T20:33:48.844751+0000","last_became_peered":"2024-03-16T20:33:48.844751+0000","last_unstale":"2024-03-16T20:34:09.940746+0000","last_undegraded":"2024-03-16T20:34:09.940746+0000","last_fullsized":"2024-03-16T20:34:09.940746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:41.148435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874856+0000","last_change":"2024-03-16T20:33:49.857181+0000","last_active":"2024-03-16T20:33:54.874856+0000","last_peered":"2024-03-16T20:33:54.874856+0000","last_clean":"2024-03-16T20:33:54.874856+0000","last_became_active":"2024-03-16T20:33:49.856866+0000","last_became_peered":"2024-03-16T20:33:49.856866+0000","last_unstale":"2024-03-16T20:33:54.874856+0000","last_undegraded":"2024-03-16T20:33:54.874856+0000","last_fullsized":"2024-03-16T20:33:54.874856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.006531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110225+0000","last_change":"2024-03-16T20:33:53.879419+0000","last_active":"2024-03-16T20:33:55.110225+0000","last_peered":"2024-03-16T20:33:55.110225+0000","last_clean":"2024-03-16T20:33:55.110225+0000","last_became_active":"2024-03-16T20:33:53.878590+0000","last_became_peered":"2024-03-16T20:33:53.878590+0000","last_unstale":"2024-03-16T20:33:55.110225+0000","last_undegraded":"2024-03-16T20:33:55.110225+0000","last_fullsized":"2024-03-16T20:33:55.110225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:06:34.552834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.925793+0000","last_change":"2024-03-16T20:33:48.845122+0000","last_active":"2024-03-16T20:34:08.925793+0000","last_peered":"2024-03-16T20:34:08.925793+0000","last_clean":"2024-03-16T20:34:08.925793+0000","last_became_active":"2024-03-16T20:33:48.844109+0000","last_became_peered":"2024-03-16T20:33:48.844109+0000","last_unstale":"2024-03-16T20:34:08.925793+0000","last_undegraded":"2024-03-16T20:34:08.925793+0000","last_fullsized":"2024-03-16T20:34:08.925793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:46.911439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109914+0000","last_change":"2024-03-16T20:33:49.850398+0000","last_active":"2024-03-16T20:33:55.109914+0000","last_peered":"2024-03-16T20:33:55.109914+0000","last_clean":"2024-03-16T20:33:55.109914+0000","last_became_active":"2024-03-16T20:33:49.850059+0000","last_became_peered":"2024-03-16T20:33:49.850059+0000","last_unstale":"2024-03-16T20:33:55.109914+0000","last_undegraded":"2024-03-16T20:33:55.109914+0000","last_fullsized":"2024-03-16T20:33:55.109914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:02.193759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110091+0000","last_change":"2024-03-16T20:33:53.890035+0000","last_active":"2024-03-16T20:33:55.110091+0000","last_peered":"2024-03-16T20:33:55.110091+0000","last_clean":"2024-03-16T20:33:55.110091+0000","last_became_active":"2024-03-16T20:33:53.889468+0000","last_became_peered":"2024-03-16T20:33:53.889468+0000","last_unstale":"2024-03-16T20:33:55.110091+0000","last_undegraded":"2024-03-16T20:33:55.110091+0000","last_fullsized":"2024-03-16T20:33:55.110091+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:47.125880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.934594+0000","last_change":"2024-03-16T20:33:48.850165+0000","last_active":"2024-03-16T20:34:08.934594+0000","last_peered":"2024-03-16T20:34:08.934594+0000","last_clean":"2024-03-16T20:34:08.934594+0000","last_became_active":"2024-03-16T20:33:48.849826+0000","last_became_peered":"2024-03-16T20:33:48.849826+0000","last_unstale":"2024-03-16T20:34:08.934594+0000","last_undegraded":"2024-03-16T20:34:08.934594+0000","last_fullsized":"2024-03-16T20:34:08.934594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:23.790227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874703+0000","last_change":"2024-03-16T20:33:49.857506+0000","last_active":"2024-03-16T20:33:54.874703+0000","last_peered":"2024-03-16T20:33:54.874703+0000","last_clean":"2024-03-16T20:33:54.874703+0000","last_became_active":"2024-03-16T20:33:49.856343+0000","last_became_peered":"2024-03-16T20:33:49.856343+0000","last_unstale":"2024-03-16T20:33:54.874703+0000","last_undegraded":"2024-03-16T20:33:54.874703+0000","last_fullsized":"2024-03-16T20:33:54.874703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:18:02.747518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156667+0000","last_change":"2024-03-16T20:33:53.884943+0000","last_active":"2024-03-16T20:33:55.156667+0000","last_peered":"2024-03-16T20:33:55.156667+0000","last_clean":"2024-03-16T20:33:55.156667+0000","last_became_active":"2024-03-16T20:33:53.884527+0000","last_became_peered":"2024-03-16T20:33:53.884527+0000","last_unstale":"2024-03-16T20:33:55.156667+0000","last_undegraded":"2024-03-16T20:33:55.156667+0000","last_fullsized":"2024-03-16T20:33:55.156667+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:26:07.594433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.726374+0000","last_change":"2024-03-16T20:33:48.849773+0000","last_active":"2024-03-16T20:34:08.726374+0000","last_peered":"2024-03-16T20:34:08.726374+0000","last_clean":"2024-03-16T20:34:08.726374+0000","last_became_active":"2024-03-16T20:33:48.848736+0000","last_became_peered":"2024-03-16T20:33:48.848736+0000","last_unstale":"2024-03-16T20:34:08.726374+0000","last_undegraded":"2024-03-16T20:34:08.726374+0000","last_fullsized":"2024-03-16T20:34:08.726374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:54:37.017119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110966+0000","last_change":"2024-03-16T20:33:49.853446+0000","last_active":"2024-03-16T20:33:55.110966+0000","last_peered":"2024-03-16T20:33:55.110966+0000","last_clean":"2024-03-16T20:33:55.110966+0000","last_became_active":"2024-03-16T20:33:49.851886+0000","last_became_peered":"2024-03-16T20:33:49.851886+0000","last_unstale":"2024-03-16T20:33:55.110966+0000","last_undegraded":"2024-03-16T20:33:55.110966+0000","last_fullsized":"2024-03-16T20:33:55.110966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:39.675229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110675+0000","last_change":"2024-03-16T20:33:53.890388+0000","last_active":"2024-03-16T20:33:55.110675+0000","last_peered":"2024-03-16T20:33:55.110675+0000","last_clean":"2024-03-16T20:33:55.110675+0000","last_became_active":"2024-03-16T20:33:53.889966+0000","last_became_peered":"2024-03-16T20:33:53.889966+0000","last_unstale":"2024-03-16T20:33:55.110675+0000","last_undegraded":"2024-03-16T20:33:55.110675+0000","last_fullsized":"2024-03-16T20:33:55.110675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:21.475392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712797+0000","last_change":"2024-03-16T20:33:48.841524+0000","last_active":"2024-03-16T20:34:07.712797+0000","last_peered":"2024-03-16T20:34:07.712797+0000","last_clean":"2024-03-16T20:34:07.712797+0000","last_became_active":"2024-03-16T20:33:48.841239+0000","last_became_peered":"2024-03-16T20:33:48.841239+0000","last_unstale":"2024-03-16T20:34:07.712797+0000","last_undegraded":"2024-03-16T20:34:07.712797+0000","last_fullsized":"2024-03-16T20:34:07.712797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:42.206113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109788+0000","last_change":"2024-03-16T20:33:49.856131+0000","last_active":"2024-03-16T20:33:55.109788+0000","last_peered":"2024-03-16T20:33:55.109788+0000","last_clean":"2024-03-16T20:33:55.109788+0000","last_became_active":"2024-03-16T20:33:49.855819+0000","last_became_peered":"2024-03-16T20:33:49.855819+0000","last_unstale":"2024-03-16T20:33:55.109788+0000","last_undegraded":"2024-03-16T20:33:55.109788+0000","last_fullsized":"2024-03-16T20:33:55.109788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:22.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110373+0000","last_change":"2024-03-16T20:33:53.888159+0000","last_active":"2024-03-16T20:33:55.110373+0000","last_peered":"2024-03-16T20:33:55.110373+0000","last_clean":"2024-03-16T20:33:55.110373+0000","last_became_active":"2024-03-16T20:33:53.887777+0000","last_became_peered":"2024-03-16T20:33:53.887777+0000","last_unstale":"2024-03-16T20:33:55.110373+0000","last_undegraded":"2024-03-16T20:33:55.110373+0000","last_fullsized":"2024-03-16T20:33:55.110373+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:26:17.021957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:10.929298+0000","last_change":"2024-03-16T20:33:48.846649+0000","last_active":"2024-03-16T20:34:10.929298+0000","last_peered":"2024-03-16T20:34:10.929298+0000","last_clean":"2024-03-16T20:34:10.929298+0000","last_became_active":"2024-03-16T20:33:48.846344+0000","last_became_peered":"2024-03-16T20:33:48.846344+0000","last_unstale":"2024-03-16T20:34:10.929298+0000","last_undegraded":"2024-03-16T20:34:10.929298+0000","last_fullsized":"2024-03-16T20:34:10.929298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:54:24.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871958+0000","last_change":"2024-03-16T20:33:49.866087+0000","last_active":"2024-03-16T20:33:54.871958+0000","last_peered":"2024-03-16T20:33:54.871958+0000","last_clean":"2024-03-16T20:33:54.871958+0000","last_became_active":"2024-03-16T20:33:49.865722+0000","last_became_peered":"2024-03-16T20:33:49.865722+0000","last_unstale":"2024-03-16T20:33:54.871958+0000","last_undegraded":"2024-03-16T20:33:54.871958+0000","last_fullsized":"2024-03-16T20:33:54.871958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:04:09.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109737+0000","last_change":"2024-03-16T20:33:53.890208+0000","last_active":"2024-03-16T20:33:55.109737+0000","last_peered":"2024-03-16T20:33:55.109737+0000","last_clean":"2024-03-16T20:33:55.109737+0000","last_became_active":"2024-03-16T20:33:53.889807+0000","last_became_peered":"2024-03-16T20:33:53.889807+0000","last_unstale":"2024-03-16T20:33:55.109737+0000","last_undegraded":"2024-03-16T20:33:55.109737+0000","last_fullsized":"2024-03-16T20:33:55.109737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:30:53.439687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712990+0000","last_change":"2024-03-16T20:33:48.851610+0000","last_active":"2024-03-16T20:34:07.712990+0000","last_peered":"2024-03-16T20:34:07.712990+0000","last_clean":"2024-03-16T20:34:07.712990+0000","last_became_active":"2024-03-16T20:33:48.851054+0000","last_became_peered":"2024-03-16T20:33:48.851054+0000","last_unstale":"2024-03-16T20:34:07.712990+0000","last_undegraded":"2024-03-16T20:34:07.712990+0000","last_fullsized":"2024-03-16T20:34:07.712990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:58:03.733318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111308+0000","last_change":"2024-03-16T20:33:49.856534+0000","last_active":"2024-03-16T20:33:55.111308+0000","last_peered":"2024-03-16T20:33:55.111308+0000","last_clean":"2024-03-16T20:33:55.111308+0000","last_became_active":"2024-03-16T20:33:49.855957+0000","last_became_peered":"2024-03-16T20:33:49.855957+0000","last_unstale":"2024-03-16T20:33:55.111308+0000","last_undegraded":"2024-03-16T20:33:55.111308+0000","last_fullsized":"2024-03-16T20:33:55.111308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:07.035334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.112291+0000","last_change":"2024-03-16T20:33:53.892149+0000","last_active":"2024-03-16T20:33:55.112291+0000","last_peered":"2024-03-16T20:33:55.112291+0000","last_clean":"2024-03-16T20:33:55.112291+0000","last_became_active":"2024-03-16T20:33:53.891730+0000","last_became_peered":"2024-03-16T20:33:53.891730+0000","last_unstale":"2024-03-16T20:33:55.112291+0000","last_undegraded":"2024-03-16T20:33:55.112291+0000","last_fullsized":"2024-03-16T20:33:55.112291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:59.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.926374+0000","last_change":"2024-03-16T20:33:48.849909+0000","last_active":"2024-03-16T20:34:08.926374+0000","last_peered":"2024-03-16T20:34:08.926374+0000","last_clean":"2024-03-16T20:34:08.926374+0000","last_became_active":"2024-03-16T20:33:48.848645+0000","last_became_peered":"2024-03-16T20:33:48.848645+0000","last_unstale":"2024-03-16T20:34:08.926374+0000","last_undegraded":"2024-03-16T20:34:08.926374+0000","last_fullsized":"2024-03-16T20:34:08.926374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:44.401679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871609+0000","last_change":"2024-03-16T20:33:49.853100+0000","last_active":"2024-03-16T20:33:54.871609+0000","last_peered":"2024-03-16T20:33:54.871609+0000","last_clean":"2024-03-16T20:33:54.871609+0000","last_became_active":"2024-03-16T20:33:49.852743+0000","last_became_peered":"2024-03-16T20:33:49.852743+0000","last_unstale":"2024-03-16T20:33:54.871609+0000","last_undegraded":"2024-03-16T20:33:54.871609+0000","last_fullsized":"2024-03-16T20:33:54.871609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:42.664026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110319+0000","last_change":"2024-03-16T20:33:53.881626+0000","last_active":"2024-03-16T20:33:55.110319+0000","last_peered":"2024-03-16T20:33:55.110319+0000","last_clean":"2024-03-16T20:33:55.110319+0000","last_became_active":"2024-03-16T20:33:53.881205+0000","last_became_peered":"2024-03-16T20:33:53.881205+0000","last_unstale":"2024-03-16T20:33:55.110319+0000","last_undegraded":"2024-03-16T20:33:55.110319+0000","last_fullsized":"2024-03-16T20:33:55.110319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:19.525953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'21","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:11.355167+0000","last_change":"2024-03-16T20:33:48.848490+0000","last_active":"2024-03-16T20:34:11.355167+0000","last_peered":"2024-03-16T20:34:11.355167+0000","last_clean":"2024-03-16T20:34:11.355167+0000","last_became_active":"2024-03-16T20:33:48.848152+0000","last_became_peered":"2024-03-16T20:33:48.848152+0000","last_unstale":"2024-03-16T20:34:11.355167+0000","last_undegraded":"2024-03-16T20:34:11.355167+0000","last_fullsized":"2024-03-16T20:34:11.355167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:43:19.908178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":23,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110228+0000","last_change":"2024-03-16T20:33:49.851269+0000","last_active":"2024-03-16T20:33:55.110228+0000","last_peered":"2024-03-16T20:33:55.110228+0000","last_clean":"2024-03-16T20:33:55.110228+0000","last_became_active":"2024-03-16T20:33:49.850963+0000","last_became_peered":"2024-03-16T20:33:49.850963+0000","last_unstale":"2024-03-16T20:33:55.110228+0000","last_undegraded":"2024-03-16T20:33:55.110228+0000","last_fullsized":"2024-03-16T20:33:55.110228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:26:01.312043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.875099+0000","last_change":"2024-03-16T20:33:53.889419+0000","last_active":"2024-03-16T20:33:54.875099+0000","last_peered":"2024-03-16T20:33:54.875099+0000","last_clean":"2024-03-16T20:33:54.875099+0000","last_became_active":"2024-03-16T20:33:53.889016+0000","last_became_peered":"2024-03-16T20:33:53.889016+0000","last_unstale":"2024-03-16T20:33:54.875099+0000","last_undegraded":"2024-03-16T20:33:54.875099+0000","last_fullsized":"2024-03-16T20:33:54.875099+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:32:58.333441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940274+0000","last_change":"2024-03-16T20:33:48.855070+0000","last_active":"2024-03-16T20:34:09.940274+0000","last_peered":"2024-03-16T20:34:09.940274+0000","last_clean":"2024-03-16T20:34:09.940274+0000","last_became_active":"2024-03-16T20:33:48.854669+0000","last_became_peered":"2024-03-16T20:33:48.854669+0000","last_unstale":"2024-03-16T20:34:09.940274+0000","last_undegraded":"2024-03-16T20:34:09.940274+0000","last_fullsized":"2024-03-16T20:34:09.940274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:38.229641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873073+0000","last_change":"2024-03-16T20:33:49.850186+0000","last_active":"2024-03-16T20:33:54.873073+0000","last_peered":"2024-03-16T20:33:54.873073+0000","last_clean":"2024-03-16T20:33:54.873073+0000","last_became_active":"2024-03-16T20:33:49.849811+0000","last_became_peered":"2024-03-16T20:33:49.849811+0000","last_unstale":"2024-03-16T20:33:54.873073+0000","last_undegraded":"2024-03-16T20:33:54.873073+0000","last_fullsized":"2024-03-16T20:33:54.873073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:13:03.399984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156428+0000","last_change":"2024-03-16T20:33:53.890030+0000","last_active":"2024-03-16T20:33:55.156428+0000","last_peered":"2024-03-16T20:33:55.156428+0000","last_clean":"2024-03-16T20:33:55.156428+0000","last_became_active":"2024-03-16T20:33:53.889602+0000","last_became_peered":"2024-03-16T20:33:53.889602+0000","last_unstale":"2024-03-16T20:33:55.156428+0000","last_undegraded":"2024-03-16T20:33:55.156428+0000","last_fullsized":"2024-03-16T20:33:55.156428+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:04.319150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.711864+0000","last_change":"2024-03-16T20:33:48.840422+0000","last_active":"2024-03-16T20:34:07.711864+0000","last_peered":"2024-03-16T20:34:07.711864+0000","last_clean":"2024-03-16T20:34:07.711864+0000","last_became_active":"2024-03-16T20:33:48.840125+0000","last_became_peered":"2024-03-16T20:33:48.840125+0000","last_unstale":"2024-03-16T20:34:07.711864+0000","last_undegraded":"2024-03-16T20:34:07.711864+0000","last_fullsized":"2024-03-16T20:34:07.711864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:05.397252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110638+0000","last_change":"2024-03-16T20:33:49.862475+0000","last_active":"2024-03-16T20:33:55.110638+0000","last_peered":"2024-03-16T20:33:55.110638+0000","last_clean":"2024-03-16T20:33:55.110638+0000","last_became_active":"2024-03-16T20:33:49.862032+0000","last_became_peered":"2024-03-16T20:33:49.862032+0000","last_unstale":"2024-03-16T20:33:55.110638+0000","last_undegraded":"2024-03-16T20:33:55.110638+0000","last_fullsized":"2024-03-16T20:33:55.110638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:16:53.319352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110920+0000","last_change":"2024-03-16T20:33:53.884019+0000","last_active":"2024-03-16T20:33:55.110920+0000","last_peered":"2024-03-16T20:33:55.110920+0000","last_clean":"2024-03-16T20:33:55.110920+0000","last_became_active":"2024-03-16T20:33:53.883608+0000","last_became_peered":"2024-03-16T20:33:53.883608+0000","last_unstale":"2024-03-16T20:33:55.110920+0000","last_undegraded":"2024-03-16T20:33:55.110920+0000","last_fullsized":"2024-03-16T20:33:55.110920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:58.689682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.932185+0000","last_change":"2024-03-16T20:33:48.848145+0000","last_active":"2024-03-16T20:34:08.932185+0000","last_peered":"2024-03-16T20:34:08.932185+0000","last_clean":"2024-03-16T20:34:08.932185+0000","last_became_active":"2024-03-16T20:33:48.847754+0000","last_became_peered":"2024-03-16T20:33:48.847754+0000","last_unstale":"2024-03-16T20:34:08.932185+0000","last_undegraded":"2024-03-16T20:34:08.932185+0000","last_fullsized":"2024-03-16T20:34:08.932185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:59.967346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110207+0000","last_change":"2024-03-16T20:33:49.851307+0000","last_active":"2024-03-16T20:33:55.110207+0000","last_peered":"2024-03-16T20:33:55.110207+0000","last_clean":"2024-03-16T20:33:55.110207+0000","last_became_active":"2024-03-16T20:33:49.850930+0000","last_became_peered":"2024-03-16T20:33:49.850930+0000","last_unstale":"2024-03-16T20:33:55.110207+0000","last_undegraded":"2024-03-16T20:33:55.110207+0000","last_fullsized":"2024-03-16T20:33:55.110207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:23:54.528895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873710+0000","last_change":"2024-03-16T20:33:53.893149+0000","last_active":"2024-03-16T20:33:54.873710+0000","last_peered":"2024-03-16T20:33:54.873710+0000","last_clean":"2024-03-16T20:33:54.873710+0000","last_became_active":"2024-03-16T20:33:53.892532+0000","last_became_peered":"2024-03-16T20:33:53.892532+0000","last_unstale":"2024-03-16T20:33:54.873710+0000","last_undegraded":"2024-03-16T20:33:54.873710+0000","last_fullsized":"2024-03-16T20:33:54.873710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:09.408974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.935261+0000","last_change":"2024-03-16T20:33:48.849788+0000","last_active":"2024-03-16T20:34:08.935261+0000","last_peered":"2024-03-16T20:34:08.935261+0000","last_clean":"2024-03-16T20:34:08.935261+0000","last_became_active":"2024-03-16T20:33:48.849360+0000","last_became_peered":"2024-03-16T20:33:48.849360+0000","last_unstale":"2024-03-16T20:34:08.935261+0000","last_undegraded":"2024-03-16T20:34:08.935261+0000","last_fullsized":"2024-03-16T20:34:08.935261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:29.026216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874438+0000","last_change":"2024-03-16T20:33:49.856702+0000","last_active":"2024-03-16T20:33:54.874438+0000","last_peered":"2024-03-16T20:33:54.874438+0000","last_clean":"2024-03-16T20:33:54.874438+0000","last_became_active":"2024-03-16T20:33:49.856373+0000","last_became_peered":"2024-03-16T20:33:49.856373+0000","last_unstale":"2024-03-16T20:33:54.874438+0000","last_undegraded":"2024-03-16T20:33:54.874438+0000","last_fullsized":"2024-03-16T20:33:54.874438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:31:40.093440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874500+0000","last_change":"2024-03-16T20:33:53.884193+0000","last_active":"2024-03-16T20:33:54.874500+0000","last_peered":"2024-03-16T20:33:54.874500+0000","last_clean":"2024-03-16T20:33:54.874500+0000","last_became_active":"2024-03-16T20:33:53.883763+0000","last_became_peered":"2024-03-16T20:33:53.883763+0000","last_unstale":"2024-03-16T20:33:54.874500+0000","last_undegraded":"2024-03-16T20:33:54.874500+0000","last_fullsized":"2024-03-16T20:33:54.874500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:50:20.969662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.923969+0000","last_change":"2024-03-16T20:33:48.851166+0000","last_active":"2024-03-16T20:34:08.923969+0000","last_peered":"2024-03-16T20:34:08.923969+0000","last_clean":"2024-03-16T20:34:08.923969+0000","last_became_active":"2024-03-16T20:33:48.850843+0000","last_became_peered":"2024-03-16T20:33:48.850843+0000","last_unstale":"2024-03-16T20:34:08.923969+0000","last_undegraded":"2024-03-16T20:34:08.923969+0000","last_fullsized":"2024-03-16T20:34:08.923969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:09.142942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156599+0000","last_change":"2024-03-16T20:33:49.860038+0000","last_active":"2024-03-16T20:33:55.156599+0000","last_peered":"2024-03-16T20:33:55.156599+0000","last_clean":"2024-03-16T20:33:55.156599+0000","last_became_active":"2024-03-16T20:33:49.859715+0000","last_became_peered":"2024-03-16T20:33:49.859715+0000","last_unstale":"2024-03-16T20:33:55.156599+0000","last_undegraded":"2024-03-16T20:33:55.156599+0000","last_fullsized":"2024-03-16T20:33:55.156599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:27:32.800166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109990+0000","last_change":"2024-03-16T20:33:53.930129+0000","last_active":"2024-03-16T20:33:55.109990+0000","last_peered":"2024-03-16T20:33:55.109990+0000","last_clean":"2024-03-16T20:33:55.109990+0000","last_became_active":"2024-03-16T20:33:53.929655+0000","last_became_peered":"2024-03-16T20:33:53.929655+0000","last_unstale":"2024-03-16T20:33:55.109990+0000","last_undegraded":"2024-03-16T20:33:55.109990+0000","last_fullsized":"2024-03-16T20:33:55.109990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:46.422720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110623+0000","last_change":"2024-03-16T20:33:48.849089+0000","last_active":"2024-03-16T20:33:55.110623+0000","last_peered":"2024-03-16T20:33:55.110623+0000","last_clean":"2024-03-16T20:33:55.110623+0000","last_became_active":"2024-03-16T20:33:48.848719+0000","last_became_peered":"2024-03-16T20:33:48.848719+0000","last_unstale":"2024-03-16T20:33:55.110623+0000","last_undegraded":"2024-03-16T20:33:55.110623+0000","last_fullsized":"2024-03-16T20:33:55.110623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:31:23.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111549+0000","last_change":"2024-03-16T20:33:49.853247+0000","last_active":"2024-03-16T20:33:55.111549+0000","last_peered":"2024-03-16T20:33:55.111549+0000","last_clean":"2024-03-16T20:33:55.111549+0000","last_became_active":"2024-03-16T20:33:49.852870+0000","last_became_peered":"2024-03-16T20:33:49.852870+0000","last_unstale":"2024-03-16T20:33:55.111549+0000","last_undegraded":"2024-03-16T20:33:55.111549+0000","last_fullsized":"2024-03-16T20:33:55.111549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:56.365484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873655+0000","last_change":"2024-03-16T20:33:53.887350+0000","last_active":"2024-03-16T20:33:54.873655+0000","last_peered":"2024-03-16T20:33:54.873655+0000","last_clean":"2024-03-16T20:33:54.873655+0000","last_became_active":"2024-03-16T20:33:53.886782+0000","last_became_peered":"2024-03-16T20:33:53.886782+0000","last_unstale":"2024-03-16T20:33:54.873655+0000","last_undegraded":"2024-03-16T20:33:54.873655+0000","last_fullsized":"2024-03-16T20:33:54.873655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:00:39.898430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110702+0000","last_change":"2024-03-16T20:33:48.851543+0000","last_active":"2024-03-16T20:33:55.110702+0000","last_peered":"2024-03-16T20:33:55.110702+0000","last_clean":"2024-03-16T20:33:55.110702+0000","last_became_active":"2024-03-16T20:33:48.851196+0000","last_became_peered":"2024-03-16T20:33:48.851196+0000","last_unstale":"2024-03-16T20:33:55.110702+0000","last_undegraded":"2024-03-16T20:33:55.110702+0000","last_fullsized":"2024-03-16T20:33:55.110702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:06.309259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111826+0000","last_change":"2024-03-16T20:33:49.856021+0000","last_active":"2024-03-16T20:33:55.111826+0000","last_peered":"2024-03-16T20:33:55.111826+0000","last_clean":"2024-03-16T20:33:55.111826+0000","last_became_active":"2024-03-16T20:33:49.854130+0000","last_became_peered":"2024-03-16T20:33:49.854130+0000","last_unstale":"2024-03-16T20:33:55.111826+0000","last_undegraded":"2024-03-16T20:33:55.111826+0000","last_fullsized":"2024-03-16T20:33:55.111826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:29:39.970391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873619+0000","last_change":"2024-03-16T20:33:53.930094+0000","last_active":"2024-03-16T20:33:54.873619+0000","last_peered":"2024-03-16T20:33:54.873619+0000","last_clean":"2024-03-16T20:33:54.873619+0000","last_became_active":"2024-03-16T20:33:53.929613+0000","last_became_peered":"2024-03-16T20:33:53.929613+0000","last_unstale":"2024-03-16T20:33:54.873619+0000","last_undegraded":"2024-03-16T20:33:54.873619+0000","last_fullsized":"2024-03-16T20:33:54.873619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:30.680295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874565+0000","last_change":"2024-03-16T20:33:49.857949+0000","last_active":"2024-03-16T20:33:54.874565+0000","last_peered":"2024-03-16T20:33:54.874565+0000","last_clean":"2024-03-16T20:33:54.874565+0000","last_became_active":"2024-03-16T20:33:49.857642+0000","last_became_peered":"2024-03-16T20:33:49.857642+0000","last_unstale":"2024-03-16T20:33:54.874565+0000","last_undegraded":"2024-03-16T20:33:54.874565+0000","last_fullsized":"2024-03-16T20:33:54.874565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:23.040787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.928996+0000","last_change":"2024-03-16T20:33:48.856908+0000","last_active":"2024-03-16T20:34:09.928996+0000","last_peered":"2024-03-16T20:34:09.928996+0000","last_clean":"2024-03-16T20:34:09.928996+0000","last_became_active":"2024-03-16T20:33:48.856578+0000","last_became_peered":"2024-03-16T20:33:48.856578+0000","last_unstale":"2024-03-16T20:34:09.928996+0000","last_undegraded":"2024-03-16T20:34:09.928996+0000","last_fullsized":"2024-03-16T20:34:09.928996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:37.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111025+0000","last_change":"2024-03-16T20:33:53.881965+0000","last_active":"2024-03-16T20:33:55.111025+0000","last_peered":"2024-03-16T20:33:55.111025+0000","last_clean":"2024-03-16T20:33:55.111025+0000","last_became_active":"2024-03-16T20:33:53.881247+0000","last_became_peered":"2024-03-16T20:33:53.881247+0000","last_unstale":"2024-03-16T20:33:55.111025+0000","last_undegraded":"2024-03-16T20:33:55.111025+0000","last_fullsized":"2024-03-16T20:33:55.111025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:18.868143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109858+0000","last_change":"2024-03-16T20:33:49.857605+0000","last_active":"2024-03-16T20:33:55.109858+0000","last_peered":"2024-03-16T20:33:55.109858+0000","last_clean":"2024-03-16T20:33:55.109858+0000","last_became_active":"2024-03-16T20:33:49.857242+0000","last_became_peered":"2024-03-16T20:33:49.857242+0000","last_unstale":"2024-03-16T20:33:55.109858+0000","last_undegraded":"2024-03-16T20:33:55.109858+0000","last_fullsized":"2024-03-16T20:33:55.109858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:55:21.442023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.933731+0000","last_change":"2024-03-16T20:33:48.850177+0000","last_active":"2024-03-16T20:34:08.933731+0000","last_peered":"2024-03-16T20:34:08.933731+0000","last_clean":"2024-03-16T20:34:08.933731+0000","last_became_active":"2024-03-16T20:33:48.849838+0000","last_became_peered":"2024-03-16T20:33:48.849838+0000","last_unstale":"2024-03-16T20:34:08.933731+0000","last_undegraded":"2024-03-16T20:34:08.933731+0000","last_fullsized":"2024-03-16T20:34:08.933731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:24.544880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110076+0000","last_change":"2024-03-16T20:32:34.913833+0000","last_active":"2024-03-16T20:33:55.110076+0000","last_peered":"2024-03-16T20:33:55.110076+0000","last_clean":"2024-03-16T20:33:55.110076+0000","last_became_active":"2024-03-16T20:32:34.913443+0000","last_became_peered":"2024-03-16T20:32:34.913443+0000","last_unstale":"2024-03-16T20:33:55.110076+0000","last_undegraded":"2024-03-16T20:33:55.110076+0000","last_fullsized":"2024-03-16T20:33:55.110076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_clean_scrub_stamp":"2024-03-16T20:29:10.760383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:05.333386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874820+0000","last_change":"2024-03-16T20:33:53.931311+0000","last_active":"2024-03-16T20:33:54.874820+0000","last_peered":"2024-03-16T20:33:54.874820+0000","last_clean":"2024-03-16T20:33:54.874820+0000","last_became_active":"2024-03-16T20:33:53.930972+0000","last_became_peered":"2024-03-16T20:33:53.930972+0000","last_unstale":"2024-03-16T20:33:54.874820+0000","last_undegraded":"2024-03-16T20:33:54.874820+0000","last_fullsized":"2024-03-16T20:33:54.874820+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:29.776651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.872757+0000","last_change":"2024-03-16T20:33:49.838866+0000","last_active":"2024-03-16T20:33:54.872757+0000","last_peered":"2024-03-16T20:33:54.872757+0000","last_clean":"2024-03-16T20:33:54.872757+0000","last_became_active":"2024-03-16T20:33:49.838561+0000","last_became_peered":"2024-03-16T20:33:49.838561+0000","last_unstale":"2024-03-16T20:33:54.872757+0000","last_undegraded":"2024-03-16T20:33:54.872757+0000","last_fullsized":"2024-03-16T20:33:54.872757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:57:21.098078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.724982+0000","last_change":"2024-03-16T20:33:48.847098+0000","last_active":"2024-03-16T20:34:08.724982+0000","last_peered":"2024-03-16T20:34:08.724982+0000","last_clean":"2024-03-16T20:34:08.724982+0000","last_became_active":"2024-03-16T20:33:48.846399+0000","last_became_peered":"2024-03-16T20:33:48.846399+0000","last_unstale":"2024-03-16T20:34:08.724982+0000","last_undegraded":"2024-03-16T20:34:08.724982+0000","last_fullsized":"2024-03-16T20:34:08.724982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:59.126547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156422+0000","last_change":"2024-03-16T20:33:53.890952+0000","last_active":"2024-03-16T20:33:55.156422+0000","last_peered":"2024-03-16T20:33:55.156422+0000","last_clean":"2024-03-16T20:33:55.156422+0000","last_became_active":"2024-03-16T20:33:53.890473+0000","last_became_peered":"2024-03-16T20:33:53.890473+0000","last_unstale":"2024-03-16T20:33:55.156422+0000","last_undegraded":"2024-03-16T20:33:55.156422+0000","last_fullsized":"2024-03-16T20:33:55.156422+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:06.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.935486+0000","last_change":"2024-03-16T20:33:48.851220+0000","last_active":"2024-03-16T20:34:08.935486+0000","last_peered":"2024-03-16T20:34:08.935486+0000","last_clean":"2024-03-16T20:34:08.935486+0000","last_became_active":"2024-03-16T20:33:48.850927+0000","last_became_peered":"2024-03-16T20:33:48.850927+0000","last_unstale":"2024-03-16T20:34:08.935486+0000","last_undegraded":"2024-03-16T20:34:08.935486+0000","last_fullsized":"2024-03-16T20:34:08.935486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:50.105276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873629+0000","last_change":"2024-03-16T20:33:49.850272+0000","last_active":"2024-03-16T20:33:54.873629+0000","last_peered":"2024-03-16T20:33:54.873629+0000","last_clean":"2024-03-16T20:33:54.873629+0000","last_became_active":"2024-03-16T20:33:49.849913+0000","last_became_peered":"2024-03-16T20:33:49.849913+0000","last_unstale":"2024-03-16T20:33:54.873629+0000","last_undegraded":"2024-03-16T20:33:54.873629+0000","last_fullsized":"2024-03-16T20:33:54.873629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:53:18.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110943+0000","last_change":"2024-03-16T20:33:53.930421+0000","last_active":"2024-03-16T20:33:55.110943+0000","last_peered":"2024-03-16T20:33:55.110943+0000","last_clean":"2024-03-16T20:33:55.110943+0000","last_became_active":"2024-03-16T20:33:53.929957+0000","last_became_peered":"2024-03-16T20:33:53.929957+0000","last_unstale":"2024-03-16T20:33:55.110943+0000","last_undegraded":"2024-03-16T20:33:55.110943+0000","last_fullsized":"2024-03-16T20:33:55.110943+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:20.541592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.936089+0000","last_change":"2024-03-16T20:33:48.845096+0000","last_active":"2024-03-16T20:34:08.936089+0000","last_peered":"2024-03-16T20:34:08.936089+0000","last_clean":"2024-03-16T20:34:08.936089+0000","last_became_active":"2024-03-16T20:33:48.844580+0000","last_became_peered":"2024-03-16T20:33:48.844580+0000","last_unstale":"2024-03-16T20:34:08.936089+0000","last_undegraded":"2024-03-16T20:34:08.936089+0000","last_fullsized":"2024-03-16T20:34:08.936089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:56:53.857336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109948+0000","last_change":"2024-03-16T20:33:49.858488+0000","last_active":"2024-03-16T20:33:55.109948+0000","last_peered":"2024-03-16T20:33:55.109948+0000","last_clean":"2024-03-16T20:33:55.109948+0000","last_became_active":"2024-03-16T20:33:49.858163+0000","last_became_peered":"2024-03-16T20:33:49.858163+0000","last_unstale":"2024-03-16T20:33:55.109948+0000","last_undegraded":"2024-03-16T20:33:55.109948+0000","last_fullsized":"2024-03-16T20:33:55.109948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:49.399329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110376+0000","last_change":"2024-03-16T20:33:53.884240+0000","last_active":"2024-03-16T20:33:55.110376+0000","last_peered":"2024-03-16T20:33:55.110376+0000","last_clean":"2024-03-16T20:33:55.110376+0000","last_became_active":"2024-03-16T20:33:53.883864+0000","last_became_peered":"2024-03-16T20:33:53.883864+0000","last_unstale":"2024-03-16T20:33:55.110376+0000","last_undegraded":"2024-03-16T20:33:55.110376+0000","last_fullsized":"2024-03-16T20:33:55.110376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:57:56.073112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940664+0000","last_change":"2024-03-16T20:33:48.858217+0000","last_active":"2024-03-16T20:34:09.940664+0000","last_peered":"2024-03-16T20:34:09.940664+0000","last_clean":"2024-03-16T20:34:09.940664+0000","last_became_active":"2024-03-16T20:33:48.857844+0000","last_became_peered":"2024-03-16T20:33:48.857844+0000","last_unstale":"2024-03-16T20:34:09.940664+0000","last_undegraded":"2024-03-16T20:34:09.940664+0000","last_fullsized":"2024-03-16T20:34:09.940664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:09.143250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110824+0000","last_change":"2024-03-16T20:33:49.856002+0000","last_active":"2024-03-16T20:33:55.110824+0000","last_peered":"2024-03-16T20:33:55.110824+0000","last_clean":"2024-03-16T20:33:55.110824+0000","last_became_active":"2024-03-16T20:33:49.855664+0000","last_became_peered":"2024-03-16T20:33:49.855664+0000","last_unstale":"2024-03-16T20:33:55.110824+0000","last_undegraded":"2024-03-16T20:33:55.110824+0000","last_fullsized":"2024-03-16T20:33:55.110824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:55:36.502729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111392+0000","last_change":"2024-03-16T20:33:53.931291+0000","last_active":"2024-03-16T20:33:55.111392+0000","last_peered":"2024-03-16T20:33:55.111392+0000","last_clean":"2024-03-16T20:33:55.111392+0000","last_became_active":"2024-03-16T20:33:53.930863+0000","last_became_peered":"2024-03-16T20:33:53.930863+0000","last_unstale":"2024-03-16T20:33:55.111392+0000","last_undegraded":"2024-03-16T20:33:55.111392+0000","last_fullsized":"2024-03-16T20:33:55.111392+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:36:59.977054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.711835+0000","last_change":"2024-03-16T20:33:48.855853+0000","last_active":"2024-03-16T20:34:07.711835+0000","last_peered":"2024-03-16T20:34:07.711835+0000","last_clean":"2024-03-16T20:34:07.711835+0000","last_became_active":"2024-03-16T20:33:48.855539+0000","last_became_peered":"2024-03-16T20:33:48.855539+0000","last_unstale":"2024-03-16T20:34:07.711835+0000","last_undegraded":"2024-03-16T20:34:07.711835+0000","last_fullsized":"2024-03-16T20:34:07.711835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:56:48.618350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109638+0000","last_change":"2024-03-16T20:33:49.854825+0000","last_active":"2024-03-16T20:33:55.109638+0000","last_peered":"2024-03-16T20:33:55.109638+0000","last_clean":"2024-03-16T20:33:55.109638+0000","last_became_active":"2024-03-16T20:33:49.854396+0000","last_became_peered":"2024-03-16T20:33:49.854396+0000","last_unstale":"2024-03-16T20:33:55.109638+0000","last_undegraded":"2024-03-16T20:33:55.109638+0000","last_fullsized":"2024-03-16T20:33:55.109638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:31:18.133694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111224+0000","last_change":"2024-03-16T20:33:53.881554+0000","last_active":"2024-03-16T20:33:55.111224+0000","last_peered":"2024-03-16T20:33:55.111224+0000","last_clean":"2024-03-16T20:33:55.111224+0000","last_became_active":"2024-03-16T20:33:53.881030+0000","last_became_peered":"2024-03-16T20:33:53.881030+0000","last_unstale":"2024-03-16T20:33:55.111224+0000","last_undegraded":"2024-03-16T20:33:55.111224+0000","last_fullsized":"2024-03-16T20:33:55.111224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:46.499466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940896+0000","last_change":"2024-03-16T20:33:48.853138+0000","last_active":"2024-03-16T20:34:09.940896+0000","last_peered":"2024-03-16T20:34:09.940896+0000","last_clean":"2024-03-16T20:34:09.940896+0000","last_became_active":"2024-03-16T20:33:48.851501+0000","last_became_peered":"2024-03-16T20:33:48.851501+0000","last_unstale":"2024-03-16T20:34:09.940896+0000","last_undegraded":"2024-03-16T20:34:09.940896+0000","last_fullsized":"2024-03-16T20:34:09.940896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:57:40.368614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156250+0000","last_change":"2024-03-16T20:33:49.850307+0000","last_active":"2024-03-16T20:33:55.156250+0000","last_peered":"2024-03-16T20:33:55.156250+0000","last_clean":"2024-03-16T20:33:55.156250+0000","last_became_active":"2024-03-16T20:33:49.849953+0000","last_became_peered":"2024-03-16T20:33:49.849953+0000","last_unstale":"2024-03-16T20:33:55.156250+0000","last_undegraded":"2024-03-16T20:33:55.156250+0000","last_fullsized":"2024-03-16T20:33:55.156250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:36:51.020011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111483+0000","last_change":"2024-03-16T20:33:53.892370+0000","last_active":"2024-03-16T20:33:55.111483+0000","last_peered":"2024-03-16T20:33:55.111483+0000","last_clean":"2024-03-16T20:33:55.111483+0000","last_became_active":"2024-03-16T20:33:53.891885+0000","last_became_peered":"2024-03-16T20:33:53.891885+0000","last_unstale":"2024-03-16T20:33:55.111483+0000","last_undegraded":"2024-03-16T20:33:55.111483+0000","last_fullsized":"2024-03-16T20:33:55.111483+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:30.770882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.707308+0000","last_change":"2024-03-16T20:33:48.855295+0000","last_active":"2024-03-16T20:34:09.707308+0000","last_peered":"2024-03-16T20:34:09.707308+0000","last_clean":"2024-03-16T20:34:09.707308+0000","last_became_active":"2024-03-16T20:33:48.854970+0000","last_became_peered":"2024-03-16T20:33:48.854970+0000","last_unstale":"2024-03-16T20:34:09.707308+0000","last_undegraded":"2024-03-16T20:34:09.707308+0000","last_fullsized":"2024-03-16T20:34:09.707308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:28.281615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110410+0000","last_change":"2024-03-16T20:33:49.861637+0000","last_active":"2024-03-16T20:33:55.110410+0000","last_peered":"2024-03-16T20:33:55.110410+0000","last_clean":"2024-03-16T20:33:55.110410+0000","last_became_active":"2024-03-16T20:33:49.861284+0000","last_became_peered":"2024-03-16T20:33:49.861284+0000","last_unstale":"2024-03-16T20:33:55.110410+0000","last_undegraded":"2024-03-16T20:33:55.110410+0000","last_fullsized":"2024-03-16T20:33:55.110410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.937928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111442+0000","last_change":"2024-03-16T20:33:53.931633+0000","last_active":"2024-03-16T20:33:55.111442+0000","last_peered":"2024-03-16T20:33:55.111442+0000","last_clean":"2024-03-16T20:33:55.111442+0000","last_became_active":"2024-03-16T20:33:53.930633+0000","last_became_peered":"2024-03-16T20:33:53.930633+0000","last_unstale":"2024-03-16T20:33:55.111442+0000","last_undegraded":"2024-03-16T20:33:55.111442+0000","last_fullsized":"2024-03-16T20:33:55.111442+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:33:57.356317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712199+0000","last_change":"2024-03-16T20:33:48.844925+0000","last_active":"2024-03-16T20:34:07.712199+0000","last_peered":"2024-03-16T20:34:07.712199+0000","last_clean":"2024-03-16T20:34:07.712199+0000","last_became_active":"2024-03-16T20:33:48.844569+0000","last_became_peered":"2024-03-16T20:33:48.844569+0000","last_unstale":"2024-03-16T20:34:07.712199+0000","last_undegraded":"2024-03-16T20:34:07.712199+0000","last_fullsized":"2024-03-16T20:34:07.712199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:04.430554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110846+0000","last_change":"2024-03-16T20:33:49.859085+0000","last_active":"2024-03-16T20:33:55.110846+0000","last_peered":"2024-03-16T20:33:55.110846+0000","last_clean":"2024-03-16T20:33:55.110846+0000","last_became_active":"2024-03-16T20:33:49.858637+0000","last_became_peered":"2024-03-16T20:33:49.858637+0000","last_unstale":"2024-03-16T20:33:55.110846+0000","last_undegraded":"2024-03-16T20:33:55.110846+0000","last_fullsized":"2024-03-16T20:33:55.110846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:22:17.020669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871355+0000","last_change":"2024-03-16T20:33:53.891866+0000","last_active":"2024-03-16T20:33:54.871355+0000","last_peered":"2024-03-16T20:33:54.871355+0000","last_clean":"2024-03-16T20:33:54.871355+0000","last_became_active":"2024-03-16T20:33:53.891424+0000","last_became_peered":"2024-03-16T20:33:53.891424+0000","last_unstale":"2024-03-16T20:33:54.871355+0000","last_undegraded":"2024-03-16T20:33:54.871355+0000","last_fullsized":"2024-03-16T20:33:54.871355+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.932053+0000","last_change":"2024-03-16T20:33:48.856124+0000","last_active":"2024-03-16T20:34:08.932053+0000","last_peered":"2024-03-16T20:34:08.932053+0000","last_clean":"2024-03-16T20:34:08.932053+0000","last_became_active":"2024-03-16T20:33:48.855813+0000","last_became_peered":"2024-03-16T20:33:48.855813+0000","last_unstale":"2024-03-16T20:34:08.932053+0000","last_undegraded":"2024-03-16T20:34:08.932053+0000","last_fullsized":"2024-03-16T20:34:08.932053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:55.122457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111209+0000","last_change":"2024-03-16T20:33:49.857479+0000","last_active":"2024-03-16T20:33:55.111209+0000","last_peered":"2024-03-16T20:33:55.111209+0000","last_clean":"2024-03-16T20:33:55.111209+0000","last_became_active":"2024-03-16T20:33:49.856121+0000","last_became_peered":"2024-03-16T20:33:49.856121+0000","last_unstale":"2024-03-16T20:33:55.111209+0000","last_undegraded":"2024-03-16T20:33:55.111209+0000","last_fullsized":"2024-03-16T20:33:55.111209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:08:55.613977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110613+0000","last_change":"2024-03-16T20:33:53.886802+0000","last_active":"2024-03-16T20:33:55.110613+0000","last_peered":"2024-03-16T20:33:55.110613+0000","last_clean":"2024-03-16T20:33:55.110613+0000","last_became_active":"2024-03-16T20:33:53.886430+0000","last_became_peered":"2024-03-16T20:33:53.886430+0000","last_unstale":"2024-03-16T20:33:55.110613+0000","last_undegraded":"2024-03-16T20:33:55.110613+0000","last_fullsized":"2024-03-16T20:33:55.110613+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:45.673571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.923823+0000","last_change":"2024-03-16T20:33:48.846628+0000","last_active":"2024-03-16T20:34:08.923823+0000","last_peered":"2024-03-16T20:34:08.923823+0000","last_clean":"2024-03-16T20:34:08.923823+0000","last_became_active":"2024-03-16T20:33:48.846276+0000","last_became_peered":"2024-03-16T20:33:48.846276+0000","last_unstale":"2024-03-16T20:34:08.923823+0000","last_undegraded":"2024-03-16T20:34:08.923823+0000","last_fullsized":"2024-03-16T20:34:08.923823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:24.323938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110768+0000","last_change":"2024-03-16T20:33:49.854041+0000","last_active":"2024-03-16T20:33:55.110768+0000","last_peered":"2024-03-16T20:33:55.110768+0000","last_clean":"2024-03-16T20:33:55.110768+0000","last_became_active":"2024-03-16T20:33:49.853653+0000","last_became_peered":"2024-03-16T20:33:49.853653+0000","last_unstale":"2024-03-16T20:33:55.110768+0000","last_undegraded":"2024-03-16T20:33:55.110768+0000","last_fullsized":"2024-03-16T20:33:55.110768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:59:10.498314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874298+0000","last_change":"2024-03-16T20:33:53.884292+0000","last_active":"2024-03-16T20:33:54.874298+0000","last_peered":"2024-03-16T20:33:54.874298+0000","last_clean":"2024-03-16T20:33:54.874298+0000","last_became_active":"2024-03-16T20:33:53.883778+0000","last_became_peered":"2024-03-16T20:33:53.883778+0000","last_unstale":"2024-03-16T20:33:54.874298+0000","last_undegraded":"2024-03-16T20:33:54.874298+0000","last_fullsized":"2024-03-16T20:33:54.874298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:33.877142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.941110+0000","last_change":"2024-03-16T20:33:48.845277+0000","last_active":"2024-03-16T20:34:09.941110+0000","last_peered":"2024-03-16T20:34:09.941110+0000","last_clean":"2024-03-16T20:34:09.941110+0000","last_became_active":"2024-03-16T20:33:48.844954+0000","last_became_peered":"2024-03-16T20:33:48.844954+0000","last_unstale":"2024-03-16T20:34:09.941110+0000","last_undegraded":"2024-03-16T20:34:09.941110+0000","last_fullsized":"2024-03-16T20:34:09.941110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:24:59.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111140+0000","last_change":"2024-03-16T20:33:49.856152+0000","last_active":"2024-03-16T20:33:55.111140+0000","last_peered":"2024-03-16T20:33:55.111140+0000","last_clean":"2024-03-16T20:33:55.111140+0000","last_became_active":"2024-03-16T20:33:49.855026+0000","last_became_peered":"2024-03-16T20:33:49.855026+0000","last_unstale":"2024-03-16T20:33:55.111140+0000","last_undegraded":"2024-03-16T20:33:55.111140+0000","last_fullsized":"2024-03-16T20:33:55.111140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:40.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156309+0000","last_change":"2024-03-16T20:33:49.851539+0000","last_active":"2024-03-16T20:33:55.156309+0000","last_peered":"2024-03-16T20:33:55.156309+0000","last_clean":"2024-03-16T20:33:55.156309+0000","last_became_active":"2024-03-16T20:33:49.851188+0000","last_became_peered":"2024-03-16T20:33:49.851188+0000","last_unstale":"2024-03-16T20:33:55.156309+0000","last_undegraded":"2024-03-16T20:33:55.156309+0000","last_fullsized":"2024-03-16T20:33:55.156309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:34:00.153832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110027+0000","last_change":"2024-03-16T20:33:49.845150+0000","last_active":"2024-03-16T20:33:55.110027+0000","last_peered":"2024-03-16T20:33:55.110027+0000","last_clean":"2024-03-16T20:33:55.110027+0000","last_became_active":"2024-03-16T20:33:49.844531+0000","last_became_peered":"2024-03-16T20:33:49.844531+0000","last_unstale":"2024-03-16T20:33:55.110027+0000","last_undegraded":"2024-03-16T20:33:55.110027+0000","last_fullsized":"2024-03-16T20:33:55.110027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:54.607559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109405+0000","last_change":"2024-03-16T20:33:49.853608+0000","last_active":"2024-03-16T20:33:55.109405+0000","last_peered":"2024-03-16T20:33:55.109405+0000","last_clean":"2024-03-16T20:33:55.109405+0000","last_became_active":"2024-03-16T20:33:49.852800+0000","last_became_peered":"2024-03-16T20:33:49.852800+0000","last_unstale":"2024-03-16T20:33:55.109405+0000","last_undegraded":"2024-03-16T20:33:55.109405+0000","last_fullsized":"2024-03-16T20:33:55.109405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:59.988313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110862+0000","last_change":"2024-03-16T20:33:49.863542+0000","last_active":"2024-03-16T20:33:55.110862+0000","last_peered":"2024-03-16T20:33:55.110862+0000","last_clean":"2024-03-16T20:33:55.110862+0000","last_became_active":"2024-03-16T20:33:49.863143+0000","last_became_peered":"2024-03-16T20:33:49.863143+0000","last_unstale":"2024-03-16T20:33:55.110862+0000","last_undegraded":"2024-03-16T20:33:55.110862+0000","last_fullsized":"2024-03-16T20:33:55.110862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:18:29.997097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873507+0000","last_change":"2024-03-16T20:33:49.848285+0000","last_active":"2024-03-16T20:33:54.873507+0000","last_peered":"2024-03-16T20:33:54.873507+0000","last_clean":"2024-03-16T20:33:54.873507+0000","last_became_active":"2024-03-16T20:33:49.847942+0000","last_became_peered":"2024-03-16T20:33:49.847942+0000","last_unstale":"2024-03-16T20:33:54.873507+0000","last_undegraded":"2024-03-16T20:33:54.873507+0000","last_fullsized":"2024-03-16T20:33:54.873507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:08.287296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156564+0000","last_change":"2024-03-16T20:33:49.853004+0000","last_active":"2024-03-16T20:33:55.156564+0000","last_peered":"2024-03-16T20:33:55.156564+0000","last_clean":"2024-03-16T20:33:55.156564+0000","last_became_active":"2024-03-16T20:33:49.852609+0000","last_became_peered":"2024-03-16T20:33:49.852609+0000","last_unstale":"2024-03-16T20:33:55.156564+0000","last_undegraded":"2024-03-16T20:33:55.156564+0000","last_fullsized":"2024-03-16T20:33:55.156564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:42:58.648008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109992+0000","last_change":"2024-03-16T20:33:49.857259+0000","last_active":"2024-03-16T20:33:55.109992+0000","last_peered":"2024-03-16T20:33:55.109992+0000","last_clean":"2024-03-16T20:33:55.109992+0000","last_became_active":"2024-03-16T20:33:49.856898+0000","last_became_peered":"2024-03-16T20:33:49.856898+0000","last_unstale":"2024-03-16T20:33:55.109992+0000","last_undegraded":"2024-03-16T20:33:55.109992+0000","last_fullsized":"2024-03-16T20:33:55.109992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:54.203804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110129+0000","last_change":"2024-03-16T20:33:49.860995+0000","last_active":"2024-03-16T20:33:55.110129+0000","last_peered":"2024-03-16T20:33:55.110129+0000","last_clean":"2024-03-16T20:33:55.110129+0000","last_became_active":"2024-03-16T20:33:49.860635+0000","last_became_peered":"2024-03-16T20:33:49.860635+0000","last_unstale":"2024-03-16T20:33:55.110129+0000","last_undegraded":"2024-03-16T20:33:55.110129+0000","last_fullsized":"2024-03-16T20:33:55.110129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:28:59.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874468+0000","last_change":"2024-03-16T20:33:49.854841+0000","last_active":"2024-03-16T20:33:54.874468+0000","last_peered":"2024-03-16T20:33:54.874468+0000","last_clean":"2024-03-16T20:33:54.874468+0000","last_became_active":"2024-03-16T20:33:49.854468+0000","last_became_peered":"2024-03-16T20:33:49.854468+0000","last_unstale":"2024-03-16T20:33:54.874468+0000","last_undegraded":"2024-03-16T20:33:54.874468+0000","last_fullsized":"2024-03-16T20:33:54.874468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:52.224203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110734+0000","last_change":"2024-03-16T20:33:49.854757+0000","last_active":"2024-03-16T20:33:55.110734+0000","last_peered":"2024-03-16T20:33:55.110734+0000","last_clean":"2024-03-16T20:33:55.110734+0000","last_became_active":"2024-03-16T20:33:49.852131+0000","last_became_peered":"2024-03-16T20:33:49.852131+0000","last_unstale":"2024-03-16T20:33:55.110734+0000","last_undegraded":"2024-03-16T20:33:55.110734+0000","last_fullsized":"2024-03-16T20:33:55.110734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:14.731341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871423+0000","last_change":"2024-03-16T20:33:49.863332+0000","last_active":"2024-03-16T20:33:54.871423+0000","last_peered":"2024-03-16T20:33:54.871423+0000","last_clean":"2024-03-16T20:33:54.871423+0000","last_became_active":"2024-03-16T20:33:49.862953+0000","last_became_peered":"2024-03-16T20:33:49.862953+0000","last_unstale":"2024-03-16T20:33:54.871423+0000","last_undegraded":"2024-03-16T20:33:54.871423+0000","last_fullsized":"2024-03-16T20:33:54.871423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:17.126504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873363+0000","last_change":"2024-03-16T20:33:49.849055+0000","last_active":"2024-03-16T20:33:54.873363+0000","last_peered":"2024-03-16T20:33:54.873363+0000","last_clean":"2024-03-16T20:33:54.873363+0000","last_became_active":"2024-03-16T20:33:49.848686+0000","last_became_peered":"2024-03-16T20:33:49.848686+0000","last_unstale":"2024-03-16T20:33:54.873363+0000","last_undegraded":"2024-03-16T20:33:54.873363+0000","last_fullsized":"2024-03-16T20:33:54.873363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:15.783993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156397+0000","last_change":"2024-03-16T20:33:49.860146+0000","last_active":"2024-03-16T20:33:55.156397+0000","last_peered":"2024-03-16T20:33:55.156397+0000","last_clean":"2024-03-16T20:33:55.156397+0000","last_became_active":"2024-03-16T20:33:49.859768+0000","last_became_peered":"2024-03-16T20:33:49.859768+0000","last_unstale":"2024-03-16T20:33:55.156397+0000","last_undegraded":"2024-03-16T20:33:55.156397+0000","last_fullsized":"2024-03-16T20:33:55.156397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:10.747472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286596,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28975,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":37,"num_read_kb":34,"num_write":207,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":52151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":483005,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":1079342,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":484127,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":483743,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1075165,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":483117,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":495199,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":483095,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":488819,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483435,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":483671,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1074849,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":12194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:34:13.465 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-16T20:34:13.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:12 smithi189 ceph-mon[36655]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:12 smithi189 ceph-mon[36655]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:13.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:12 smithi189 ceph-mon[36655]: pgmap v301: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-16T20:34:13.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:12 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/107274482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 -- 172.21.15.23:0/1565054382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 msgr2=0x7f5904106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 --2- 172.21.15.23:0/1565054382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904106cb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009900 tx=0x7f58f402bc30 comp rx=0 tx=0).stop 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 -- 172.21.15.23:0/1565054382 shutdown_connections 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 --2- 172.21.15.23:0/1565054382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59041071f0 0x7f5904109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 --2- 172.21.15.23:0/1565054382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904106cb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 --2- 172.21.15.23:0/1565054382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5904101f90 0x7f5904104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.596+0000 7f590c55c640 1 -- 172.21.15.23:0/1565054382 >> 172.21.15.23:0/1565054382 conn(0x7f59040fb980 msgr2=0x7f59040fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- 172.21.15.23:0/1565054382 shutdown_connections 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- 172.21.15.23:0/1565054382 wait complete. 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 Processor -- start 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- start start 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5904101f90 0x7f59041975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 0x7f590419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419f0d0 con 0x7f59041048c0 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419f240 con 0x7f59041071f0 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590c55c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590419f400 con 0x7f5904101f90 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f5909af7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f5909af7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60126/0 (socket says 172.21.15.23:60126) 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f5909af7640 1 -- 172.21.15.23:0/4263579903 learned_addr learned my addr 172.21.15.23:0/4263579903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:13.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590aaf9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 0x7f590419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.597+0000 7f590a2f8640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5904101f90 0x7f59041975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 -- 172.21.15.23:0/4263579903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5904101f90 msgr2=0x7f59041975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5904101f90 0x7f59041975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 -- 172.21.15.23:0/4263579903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 msgr2=0x7f590419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 0x7f590419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 -- 172.21.15.23:0/4263579903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4009560 con 0x7f59041048c0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f590aaf9640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 0x7f590419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f5909af7640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58f4005ab0 tx=0x7f58f4005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40043d0 con 0x7f59041048c0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58f40052d0 con 0x7f59041048c0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590419f680 con 0x7f59041048c0 2024-03-16T20:34:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4003d40 con 0x7f59041048c0 2024-03-16T20:34:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.598+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590419fb90 con 0x7f59041048c0 2024-03-16T20:34:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.599+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59040fd470 con 0x7f59041048c0 2024-03-16T20:34:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.600+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f58f4005440 con 0x7f59041048c0 2024-03-16T20:34:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.601+0000 7f58fb7fe640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 0x7f58f00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.601+0000 7f590a2f8640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 0x7f58f00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.601+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58f40efd90 con 0x7f59041048c0 2024-03-16T20:34:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.601+0000 7f590a2f8640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 0x7f58f00a3f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009bd0 tx=0x7f58fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:13.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.604+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f58f40b8d10 con 0x7f59041048c0 2024-03-16T20:34:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.781+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f590402d000 con 0x7f58f00a1ac0 2024-03-16T20:34:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.790+0000 7f58fb7fe640 1 -- 172.21.15.23:0/4263579903 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340609 (secure 0 0 0) 0x7f590402d000 con 0x7f58f00a1ac0 2024-03-16T20:34:13.799 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:13.801 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-16T20:34:13.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 msgr2=0x7f58f00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 0x7f58f00a3f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009bd0 tx=0x7f58fc005f90 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 msgr2=0x7f5904197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58f4005ab0 tx=0x7f58f4005e50 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 shutdown_connections 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f58f00a1ac0 0x7f58f00a3f80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59041071f0 0x7f590419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59041048c0 0x7f5904197b20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 --2- 172.21.15.23:0/4263579903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5904101f90 0x7f59041975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 >> 172.21.15.23:0/4263579903 conn(0x7f59040fb980 msgr2=0x7f5904102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 shutdown_connections 2024-03-16T20:34:13.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:13.794+0000 7f590c55c640 1 -- 172.21.15.23:0/4263579903 wait complete. 2024-03-16T20:34:13.810 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-16T20:34:11.944681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619343,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":163,"num_read_kb":143,"num_write":388,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373496,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568328,"statfs":{"total":1151940427776,"available":1151557967872,"internally_reserved":0,"allocated":12140544,"data_stored":7597567,"data_compressed":285306,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26565,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":37,"num_read_kb":34,"num_write":162,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156791+0000","last_change":"2024-03-16T20:33:49.858111+0000","last_active":"2024-03-16T20:33:55.156791+0000","last_peered":"2024-03-16T20:33:55.156791+0000","last_clean":"2024-03-16T20:33:55.156791+0000","last_became_active":"2024-03-16T20:33:49.857656+0000","last_became_peered":"2024-03-16T20:33:49.857656+0000","last_unstale":"2024-03-16T20:33:55.156791+0000","last_undegraded":"2024-03-16T20:33:55.156791+0000","last_fullsized":"2024-03-16T20:33:55.156791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:16:47.779595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110728+0000","last_change":"2024-03-16T20:33:49.854850+0000","last_active":"2024-03-16T20:33:55.110728+0000","last_peered":"2024-03-16T20:33:55.110728+0000","last_clean":"2024-03-16T20:33:55.110728+0000","last_became_active":"2024-03-16T20:33:49.854399+0000","last_became_peered":"2024-03-16T20:33:49.854399+0000","last_unstale":"2024-03-16T20:33:55.110728+0000","last_undegraded":"2024-03-16T20:33:55.110728+0000","last_fullsized":"2024-03-16T20:33:55.110728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:55.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111050+0000","last_change":"2024-03-16T20:33:49.858831+0000","last_active":"2024-03-16T20:33:55.111050+0000","last_peered":"2024-03-16T20:33:55.111050+0000","last_clean":"2024-03-16T20:33:55.111050+0000","last_became_active":"2024-03-16T20:33:49.858507+0000","last_became_peered":"2024-03-16T20:33:49.858507+0000","last_unstale":"2024-03-16T20:33:55.111050+0000","last_undegraded":"2024-03-16T20:33:55.111050+0000","last_fullsized":"2024-03-16T20:33:55.111050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:35.447285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111877+0000","last_change":"2024-03-16T20:33:49.852269+0000","last_active":"2024-03-16T20:33:55.111877+0000","last_peered":"2024-03-16T20:33:55.111877+0000","last_clean":"2024-03-16T20:33:55.111877+0000","last_became_active":"2024-03-16T20:33:49.851904+0000","last_became_peered":"2024-03-16T20:33:49.851904+0000","last_unstale":"2024-03-16T20:33:55.111877+0000","last_undegraded":"2024-03-16T20:33:55.111877+0000","last_fullsized":"2024-03-16T20:33:55.111877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:38.520111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109628+0000","last_change":"2024-03-16T20:33:49.852371+0000","last_active":"2024-03-16T20:33:55.109628+0000","last_peered":"2024-03-16T20:33:55.109628+0000","last_clean":"2024-03-16T20:33:55.109628+0000","last_became_active":"2024-03-16T20:33:49.852034+0000","last_became_peered":"2024-03-16T20:33:49.852034+0000","last_unstale":"2024-03-16T20:33:55.109628+0000","last_undegraded":"2024-03-16T20:33:55.109628+0000","last_fullsized":"2024-03-16T20:33:55.109628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:10.229852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110839+0000","last_change":"2024-03-16T20:33:49.855209+0000","last_active":"2024-03-16T20:33:55.110839+0000","last_peered":"2024-03-16T20:33:55.110839+0000","last_clean":"2024-03-16T20:33:55.110839+0000","last_became_active":"2024-03-16T20:33:49.854502+0000","last_became_peered":"2024-03-16T20:33:49.854502+0000","last_unstale":"2024-03-16T20:33:55.110839+0000","last_undegraded":"2024-03-16T20:33:55.110839+0000","last_fullsized":"2024-03-16T20:33:55.110839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:47.792663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110377+0000","last_change":"2024-03-16T20:33:49.852668+0000","last_active":"2024-03-16T20:33:55.110377+0000","last_peered":"2024-03-16T20:33:55.110377+0000","last_clean":"2024-03-16T20:33:55.110377+0000","last_became_active":"2024-03-16T20:33:49.852263+0000","last_became_peered":"2024-03-16T20:33:49.852263+0000","last_unstale":"2024-03-16T20:33:55.110377+0000","last_undegraded":"2024-03-16T20:33:55.110377+0000","last_fullsized":"2024-03-16T20:33:55.110377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:49.815894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110713+0000","last_change":"2024-03-16T20:33:49.854962+0000","last_active":"2024-03-16T20:33:55.110713+0000","last_peered":"2024-03-16T20:33:55.110713+0000","last_clean":"2024-03-16T20:33:55.110713+0000","last_became_active":"2024-03-16T20:33:49.853963+0000","last_became_peered":"2024-03-16T20:33:49.853963+0000","last_unstale":"2024-03-16T20:33:55.110713+0000","last_undegraded":"2024-03-16T20:33:55.110713+0000","last_fullsized":"2024-03-16T20:33:55.110713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:37:19.213607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871953+0000","last_change":"2024-03-16T20:33:49.857741+0000","last_active":"2024-03-16T20:33:54.871953+0000","last_peered":"2024-03-16T20:33:54.871953+0000","last_clean":"2024-03-16T20:33:54.871953+0000","last_became_active":"2024-03-16T20:33:49.857354+0000","last_became_peered":"2024-03-16T20:33:49.857354+0000","last_unstale":"2024-03-16T20:33:54.871953+0000","last_undegraded":"2024-03-16T20:33:54.871953+0000","last_fullsized":"2024-03-16T20:33:54.871953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:22.085460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.112211+0000","last_change":"2024-03-16T20:33:49.864008+0000","last_active":"2024-03-16T20:33:55.112211+0000","last_peered":"2024-03-16T20:33:55.112211+0000","last_clean":"2024-03-16T20:33:55.112211+0000","last_became_active":"2024-03-16T20:33:49.862779+0000","last_became_peered":"2024-03-16T20:33:49.862779+0000","last_unstale":"2024-03-16T20:33:55.112211+0000","last_undegraded":"2024-03-16T20:33:55.112211+0000","last_fullsized":"2024-03-16T20:33:55.112211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:34:23.249973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110994+0000","last_change":"2024-03-16T20:33:49.858951+0000","last_active":"2024-03-16T20:33:55.110994+0000","last_peered":"2024-03-16T20:33:55.110994+0000","last_clean":"2024-03-16T20:33:55.110994+0000","last_became_active":"2024-03-16T20:33:49.858575+0000","last_became_peered":"2024-03-16T20:33:49.858575+0000","last_unstale":"2024-03-16T20:33:55.110994+0000","last_undegraded":"2024-03-16T20:33:55.110994+0000","last_fullsized":"2024-03-16T20:33:55.110994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:07:01.030826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111765+0000","last_change":"2024-03-16T20:33:49.854399+0000","last_active":"2024-03-16T20:33:55.111765+0000","last_peered":"2024-03-16T20:33:55.111765+0000","last_clean":"2024-03-16T20:33:55.111765+0000","last_became_active":"2024-03-16T20:33:49.853937+0000","last_became_peered":"2024-03-16T20:33:49.853937+0000","last_unstale":"2024-03-16T20:33:55.111765+0000","last_undegraded":"2024-03-16T20:33:55.111765+0000","last_fullsized":"2024-03-16T20:33:55.111765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:48:00.207213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873323+0000","last_change":"2024-03-16T20:33:49.853101+0000","last_active":"2024-03-16T20:33:54.873323+0000","last_peered":"2024-03-16T20:33:54.873323+0000","last_clean":"2024-03-16T20:33:54.873323+0000","last_became_active":"2024-03-16T20:33:49.852667+0000","last_became_peered":"2024-03-16T20:33:49.852667+0000","last_unstale":"2024-03-16T20:33:54.873323+0000","last_undegraded":"2024-03-16T20:33:54.873323+0000","last_fullsized":"2024-03-16T20:33:54.873323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:17:43.047348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110331+0000","last_change":"2024-03-16T20:33:49.852907+0000","last_active":"2024-03-16T20:33:55.110331+0000","last_peered":"2024-03-16T20:33:55.110331+0000","last_clean":"2024-03-16T20:33:55.110331+0000","last_became_active":"2024-03-16T20:33:49.852552+0000","last_became_peered":"2024-03-16T20:33:49.852552+0000","last_unstale":"2024-03-16T20:33:55.110331+0000","last_undegraded":"2024-03-16T20:33:55.110331+0000","last_fullsized":"2024-03-16T20:33:55.110331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:46:34.019736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110107+0000","last_change":"2024-03-16T20:33:49.863477+0000","last_active":"2024-03-16T20:33:55.110107+0000","last_peered":"2024-03-16T20:33:55.110107+0000","last_clean":"2024-03-16T20:33:55.110107+0000","last_became_active":"2024-03-16T20:33:49.863163+0000","last_became_peered":"2024-03-16T20:33:49.863163+0000","last_unstale":"2024-03-16T20:33:55.110107+0000","last_undegraded":"2024-03-16T20:33:55.110107+0000","last_fullsized":"2024-03-16T20:33:55.110107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:46:55.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.870977+0000","last_change":"2024-03-16T20:33:49.863312+0000","last_active":"2024-03-16T20:33:54.870977+0000","last_peered":"2024-03-16T20:33:54.870977+0000","last_clean":"2024-03-16T20:33:54.870977+0000","last_became_active":"2024-03-16T20:33:49.862918+0000","last_became_peered":"2024-03-16T20:33:49.862918+0000","last_unstale":"2024-03-16T20:33:54.870977+0000","last_undegraded":"2024-03-16T20:33:54.870977+0000","last_fullsized":"2024-03-16T20:33:54.870977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:33.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156624+0000","last_change":"2024-03-16T20:33:49.858863+0000","last_active":"2024-03-16T20:33:55.156624+0000","last_peered":"2024-03-16T20:33:55.156624+0000","last_clean":"2024-03-16T20:33:55.156624+0000","last_became_active":"2024-03-16T20:33:49.858543+0000","last_became_peered":"2024-03-16T20:33:49.858543+0000","last_unstale":"2024-03-16T20:33:55.156624+0000","last_undegraded":"2024-03-16T20:33:55.156624+0000","last_fullsized":"2024-03-16T20:33:55.156624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:04.584473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110362+0000","last_change":"2024-03-16T20:33:49.857863+0000","last_active":"2024-03-16T20:33:55.110362+0000","last_peered":"2024-03-16T20:33:55.110362+0000","last_clean":"2024-03-16T20:33:55.110362+0000","last_became_active":"2024-03-16T20:33:49.857503+0000","last_became_peered":"2024-03-16T20:33:49.857503+0000","last_unstale":"2024-03-16T20:33:55.110362+0000","last_undegraded":"2024-03-16T20:33:55.110362+0000","last_fullsized":"2024-03-16T20:33:55.110362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:11.206926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110192+0000","last_change":"2024-03-16T20:33:49.849940+0000","last_active":"2024-03-16T20:33:55.110192+0000","last_peered":"2024-03-16T20:33:55.110192+0000","last_clean":"2024-03-16T20:33:55.110192+0000","last_became_active":"2024-03-16T20:33:49.849632+0000","last_became_peered":"2024-03-16T20:33:49.849632+0000","last_unstale":"2024-03-16T20:33:55.110192+0000","last_undegraded":"2024-03-16T20:33:55.110192+0000","last_fullsized":"2024-03-16T20:33:55.110192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:25.921270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109900+0000","last_change":"2024-03-16T20:33:53.882456+0000","last_active":"2024-03-16T20:33:55.109900+0000","last_peered":"2024-03-16T20:33:55.109900+0000","last_clean":"2024-03-16T20:33:55.109900+0000","last_became_active":"2024-03-16T20:33:53.882040+0000","last_became_peered":"2024-03-16T20:33:53.882040+0000","last_unstale":"2024-03-16T20:33:55.109900+0000","last_undegraded":"2024-03-16T20:33:55.109900+0000","last_fullsized":"2024-03-16T20:33:55.109900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:59.392600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.942164+0000","last_change":"2024-03-16T20:33:48.846129+0000","last_active":"2024-03-16T20:34:09.942164+0000","last_peered":"2024-03-16T20:34:09.942164+0000","last_clean":"2024-03-16T20:34:09.942164+0000","last_became_active":"2024-03-16T20:33:48.845783+0000","last_became_peered":"2024-03-16T20:33:48.845783+0000","last_unstale":"2024-03-16T20:34:09.942164+0000","last_undegraded":"2024-03-16T20:34:09.942164+0000","last_fullsized":"2024-03-16T20:34:09.942164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:46:03.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110676+0000","last_change":"2024-03-16T20:33:49.851397+0000","last_active":"2024-03-16T20:33:55.110676+0000","last_peered":"2024-03-16T20:33:55.110676+0000","last_clean":"2024-03-16T20:33:55.110676+0000","last_became_active":"2024-03-16T20:33:49.851097+0000","last_became_peered":"2024-03-16T20:33:49.851097+0000","last_unstale":"2024-03-16T20:33:55.110676+0000","last_undegraded":"2024-03-16T20:33:55.110676+0000","last_fullsized":"2024-03-16T20:33:55.110676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:57:55.165017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111379+0000","last_change":"2024-03-16T20:33:53.893593+0000","last_active":"2024-03-16T20:33:55.111379+0000","last_peered":"2024-03-16T20:33:55.111379+0000","last_clean":"2024-03-16T20:33:55.111379+0000","last_became_active":"2024-03-16T20:33:53.893101+0000","last_became_peered":"2024-03-16T20:33:53.893101+0000","last_unstale":"2024-03-16T20:33:55.111379+0000","last_undegraded":"2024-03-16T20:33:55.111379+0000","last_fullsized":"2024-03-16T20:33:55.111379+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:22:59.472049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111001+0000","last_change":"2024-03-16T20:33:48.852786+0000","last_active":"2024-03-16T20:33:55.111001+0000","last_peered":"2024-03-16T20:33:55.111001+0000","last_clean":"2024-03-16T20:33:55.111001+0000","last_became_active":"2024-03-16T20:33:48.852477+0000","last_became_peered":"2024-03-16T20:33:48.852477+0000","last_unstale":"2024-03-16T20:33:55.111001+0000","last_undegraded":"2024-03-16T20:33:55.111001+0000","last_fullsized":"2024-03-16T20:33:55.111001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:49.191312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111377+0000","last_change":"2024-03-16T20:33:49.859123+0000","last_active":"2024-03-16T20:33:55.111377+0000","last_peered":"2024-03-16T20:33:55.111377+0000","last_clean":"2024-03-16T20:33:55.111377+0000","last_became_active":"2024-03-16T20:33:49.858746+0000","last_became_peered":"2024-03-16T20:33:49.858746+0000","last_unstale":"2024-03-16T20:33:55.111377+0000","last_undegraded":"2024-03-16T20:33:55.111377+0000","last_fullsized":"2024-03-16T20:33:55.111377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:35.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111721+0000","last_change":"2024-03-16T20:33:53.891821+0000","last_active":"2024-03-16T20:33:55.111721+0000","last_peered":"2024-03-16T20:33:55.111721+0000","last_clean":"2024-03-16T20:33:55.111721+0000","last_became_active":"2024-03-16T20:33:53.891383+0000","last_became_peered":"2024-03-16T20:33:53.891383+0000","last_unstale":"2024-03-16T20:33:55.111721+0000","last_undegraded":"2024-03-16T20:33:55.111721+0000","last_fullsized":"2024-03-16T20:33:55.111721+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:39:03.784185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.942046+0000","last_change":"2024-03-16T20:33:48.845405+0000","last_active":"2024-03-16T20:34:09.942046+0000","last_peered":"2024-03-16T20:34:09.942046+0000","last_clean":"2024-03-16T20:34:09.942046+0000","last_became_active":"2024-03-16T20:33:48.845033+0000","last_became_peered":"2024-03-16T20:33:48.845033+0000","last_unstale":"2024-03-16T20:34:09.942046+0000","last_undegraded":"2024-03-16T20:34:09.942046+0000","last_fullsized":"2024-03-16T20:34:09.942046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:52.632960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871503+0000","last_change":"2024-03-16T20:33:49.864934+0000","last_active":"2024-03-16T20:33:54.871503+0000","last_peered":"2024-03-16T20:33:54.871503+0000","last_clean":"2024-03-16T20:33:54.871503+0000","last_became_active":"2024-03-16T20:33:49.864574+0000","last_became_peered":"2024-03-16T20:33:49.864574+0000","last_unstale":"2024-03-16T20:33:54.871503+0000","last_undegraded":"2024-03-16T20:33:54.871503+0000","last_fullsized":"2024-03-16T20:33:54.871503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:07.010788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111913+0000","last_change":"2024-03-16T20:33:53.890446+0000","last_active":"2024-03-16T20:33:55.111913+0000","last_peered":"2024-03-16T20:33:55.111913+0000","last_clean":"2024-03-16T20:33:55.111913+0000","last_became_active":"2024-03-16T20:33:53.889832+0000","last_became_peered":"2024-03-16T20:33:53.889832+0000","last_unstale":"2024-03-16T20:33:55.111913+0000","last_undegraded":"2024-03-16T20:33:55.111913+0000","last_fullsized":"2024-03-16T20:33:55.111913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:49.031999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940746+0000","last_change":"2024-03-16T20:33:48.845809+0000","last_active":"2024-03-16T20:34:09.940746+0000","last_peered":"2024-03-16T20:34:09.940746+0000","last_clean":"2024-03-16T20:34:09.940746+0000","last_became_active":"2024-03-16T20:33:48.844751+0000","last_became_peered":"2024-03-16T20:33:48.844751+0000","last_unstale":"2024-03-16T20:34:09.940746+0000","last_undegraded":"2024-03-16T20:34:09.940746+0000","last_fullsized":"2024-03-16T20:34:09.940746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:41.148435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874856+0000","last_change":"2024-03-16T20:33:49.857181+0000","last_active":"2024-03-16T20:33:54.874856+0000","last_peered":"2024-03-16T20:33:54.874856+0000","last_clean":"2024-03-16T20:33:54.874856+0000","last_became_active":"2024-03-16T20:33:49.856866+0000","last_became_peered":"2024-03-16T20:33:49.856866+0000","last_unstale":"2024-03-16T20:33:54.874856+0000","last_undegraded":"2024-03-16T20:33:54.874856+0000","last_fullsized":"2024-03-16T20:33:54.874856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.006531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110225+0000","last_change":"2024-03-16T20:33:53.879419+0000","last_active":"2024-03-16T20:33:55.110225+0000","last_peered":"2024-03-16T20:33:55.110225+0000","last_clean":"2024-03-16T20:33:55.110225+0000","last_became_active":"2024-03-16T20:33:53.878590+0000","last_became_peered":"2024-03-16T20:33:53.878590+0000","last_unstale":"2024-03-16T20:33:55.110225+0000","last_undegraded":"2024-03-16T20:33:55.110225+0000","last_fullsized":"2024-03-16T20:33:55.110225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:06:34.552834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.925793+0000","last_change":"2024-03-16T20:33:48.845122+0000","last_active":"2024-03-16T20:34:08.925793+0000","last_peered":"2024-03-16T20:34:08.925793+0000","last_clean":"2024-03-16T20:34:08.925793+0000","last_became_active":"2024-03-16T20:33:48.844109+0000","last_became_peered":"2024-03-16T20:33:48.844109+0000","last_unstale":"2024-03-16T20:34:08.925793+0000","last_undegraded":"2024-03-16T20:34:08.925793+0000","last_fullsized":"2024-03-16T20:34:08.925793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:46.911439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109914+0000","last_change":"2024-03-16T20:33:49.850398+0000","last_active":"2024-03-16T20:33:55.109914+0000","last_peered":"2024-03-16T20:33:55.109914+0000","last_clean":"2024-03-16T20:33:55.109914+0000","last_became_active":"2024-03-16T20:33:49.850059+0000","last_became_peered":"2024-03-16T20:33:49.850059+0000","last_unstale":"2024-03-16T20:33:55.109914+0000","last_undegraded":"2024-03-16T20:33:55.109914+0000","last_fullsized":"2024-03-16T20:33:55.109914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:02.193759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110091+0000","last_change":"2024-03-16T20:33:53.890035+0000","last_active":"2024-03-16T20:33:55.110091+0000","last_peered":"2024-03-16T20:33:55.110091+0000","last_clean":"2024-03-16T20:33:55.110091+0000","last_became_active":"2024-03-16T20:33:53.889468+0000","last_became_peered":"2024-03-16T20:33:53.889468+0000","last_unstale":"2024-03-16T20:33:55.110091+0000","last_undegraded":"2024-03-16T20:33:55.110091+0000","last_fullsized":"2024-03-16T20:33:55.110091+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:47.125880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.934594+0000","last_change":"2024-03-16T20:33:48.850165+0000","last_active":"2024-03-16T20:34:08.934594+0000","last_peered":"2024-03-16T20:34:08.934594+0000","last_clean":"2024-03-16T20:34:08.934594+0000","last_became_active":"2024-03-16T20:33:48.849826+0000","last_became_peered":"2024-03-16T20:33:48.849826+0000","last_unstale":"2024-03-16T20:34:08.934594+0000","last_undegraded":"2024-03-16T20:34:08.934594+0000","last_fullsized":"2024-03-16T20:34:08.934594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:23.790227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874703+0000","last_change":"2024-03-16T20:33:49.857506+0000","last_active":"2024-03-16T20:33:54.874703+0000","last_peered":"2024-03-16T20:33:54.874703+0000","last_clean":"2024-03-16T20:33:54.874703+0000","last_became_active":"2024-03-16T20:33:49.856343+0000","last_became_peered":"2024-03-16T20:33:49.856343+0000","last_unstale":"2024-03-16T20:33:54.874703+0000","last_undegraded":"2024-03-16T20:33:54.874703+0000","last_fullsized":"2024-03-16T20:33:54.874703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:18:02.747518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156667+0000","last_change":"2024-03-16T20:33:53.884943+0000","last_active":"2024-03-16T20:33:55.156667+0000","last_peered":"2024-03-16T20:33:55.156667+0000","last_clean":"2024-03-16T20:33:55.156667+0000","last_became_active":"2024-03-16T20:33:53.884527+0000","last_became_peered":"2024-03-16T20:33:53.884527+0000","last_unstale":"2024-03-16T20:33:55.156667+0000","last_undegraded":"2024-03-16T20:33:55.156667+0000","last_fullsized":"2024-03-16T20:33:55.156667+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:26:07.594433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.726374+0000","last_change":"2024-03-16T20:33:48.849773+0000","last_active":"2024-03-16T20:34:08.726374+0000","last_peered":"2024-03-16T20:34:08.726374+0000","last_clean":"2024-03-16T20:34:08.726374+0000","last_became_active":"2024-03-16T20:33:48.848736+0000","last_became_peered":"2024-03-16T20:33:48.848736+0000","last_unstale":"2024-03-16T20:34:08.726374+0000","last_undegraded":"2024-03-16T20:34:08.726374+0000","last_fullsized":"2024-03-16T20:34:08.726374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:54:37.017119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110966+0000","last_change":"2024-03-16T20:33:49.853446+0000","last_active":"2024-03-16T20:33:55.110966+0000","last_peered":"2024-03-16T20:33:55.110966+0000","last_clean":"2024-03-16T20:33:55.110966+0000","last_became_active":"2024-03-16T20:33:49.851886+0000","last_became_peered":"2024-03-16T20:33:49.851886+0000","last_unstale":"2024-03-16T20:33:55.110966+0000","last_undegraded":"2024-03-16T20:33:55.110966+0000","last_fullsized":"2024-03-16T20:33:55.110966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:39.675229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110675+0000","last_change":"2024-03-16T20:33:53.890388+0000","last_active":"2024-03-16T20:33:55.110675+0000","last_peered":"2024-03-16T20:33:55.110675+0000","last_clean":"2024-03-16T20:33:55.110675+0000","last_became_active":"2024-03-16T20:33:53.889966+0000","last_became_peered":"2024-03-16T20:33:53.889966+0000","last_unstale":"2024-03-16T20:33:55.110675+0000","last_undegraded":"2024-03-16T20:33:55.110675+0000","last_fullsized":"2024-03-16T20:33:55.110675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:21.475392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712797+0000","last_change":"2024-03-16T20:33:48.841524+0000","last_active":"2024-03-16T20:34:07.712797+0000","last_peered":"2024-03-16T20:34:07.712797+0000","last_clean":"2024-03-16T20:34:07.712797+0000","last_became_active":"2024-03-16T20:33:48.841239+0000","last_became_peered":"2024-03-16T20:33:48.841239+0000","last_unstale":"2024-03-16T20:34:07.712797+0000","last_undegraded":"2024-03-16T20:34:07.712797+0000","last_fullsized":"2024-03-16T20:34:07.712797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:42.206113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109788+0000","last_change":"2024-03-16T20:33:49.856131+0000","last_active":"2024-03-16T20:33:55.109788+0000","last_peered":"2024-03-16T20:33:55.109788+0000","last_clean":"2024-03-16T20:33:55.109788+0000","last_became_active":"2024-03-16T20:33:49.855819+0000","last_became_peered":"2024-03-16T20:33:49.855819+0000","last_unstale":"2024-03-16T20:33:55.109788+0000","last_undegraded":"2024-03-16T20:33:55.109788+0000","last_fullsized":"2024-03-16T20:33:55.109788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:22.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110373+0000","last_change":"2024-03-16T20:33:53.888159+0000","last_active":"2024-03-16T20:33:55.110373+0000","last_peered":"2024-03-16T20:33:55.110373+0000","last_clean":"2024-03-16T20:33:55.110373+0000","last_became_active":"2024-03-16T20:33:53.887777+0000","last_became_peered":"2024-03-16T20:33:53.887777+0000","last_unstale":"2024-03-16T20:33:55.110373+0000","last_undegraded":"2024-03-16T20:33:55.110373+0000","last_fullsized":"2024-03-16T20:33:55.110373+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:26:17.021957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:10.929298+0000","last_change":"2024-03-16T20:33:48.846649+0000","last_active":"2024-03-16T20:34:10.929298+0000","last_peered":"2024-03-16T20:34:10.929298+0000","last_clean":"2024-03-16T20:34:10.929298+0000","last_became_active":"2024-03-16T20:33:48.846344+0000","last_became_peered":"2024-03-16T20:33:48.846344+0000","last_unstale":"2024-03-16T20:34:10.929298+0000","last_undegraded":"2024-03-16T20:34:10.929298+0000","last_fullsized":"2024-03-16T20:34:10.929298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:54:24.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871958+0000","last_change":"2024-03-16T20:33:49.866087+0000","last_active":"2024-03-16T20:33:54.871958+0000","last_peered":"2024-03-16T20:33:54.871958+0000","last_clean":"2024-03-16T20:33:54.871958+0000","last_became_active":"2024-03-16T20:33:49.865722+0000","last_became_peered":"2024-03-16T20:33:49.865722+0000","last_unstale":"2024-03-16T20:33:54.871958+0000","last_undegraded":"2024-03-16T20:33:54.871958+0000","last_fullsized":"2024-03-16T20:33:54.871958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:04:09.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109737+0000","last_change":"2024-03-16T20:33:53.890208+0000","last_active":"2024-03-16T20:33:55.109737+0000","last_peered":"2024-03-16T20:33:55.109737+0000","last_clean":"2024-03-16T20:33:55.109737+0000","last_became_active":"2024-03-16T20:33:53.889807+0000","last_became_peered":"2024-03-16T20:33:53.889807+0000","last_unstale":"2024-03-16T20:33:55.109737+0000","last_undegraded":"2024-03-16T20:33:55.109737+0000","last_fullsized":"2024-03-16T20:33:55.109737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:30:53.439687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712990+0000","last_change":"2024-03-16T20:33:48.851610+0000","last_active":"2024-03-16T20:34:07.712990+0000","last_peered":"2024-03-16T20:34:07.712990+0000","last_clean":"2024-03-16T20:34:07.712990+0000","last_became_active":"2024-03-16T20:33:48.851054+0000","last_became_peered":"2024-03-16T20:33:48.851054+0000","last_unstale":"2024-03-16T20:34:07.712990+0000","last_undegraded":"2024-03-16T20:34:07.712990+0000","last_fullsized":"2024-03-16T20:34:07.712990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:58:03.733318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111308+0000","last_change":"2024-03-16T20:33:49.856534+0000","last_active":"2024-03-16T20:33:55.111308+0000","last_peered":"2024-03-16T20:33:55.111308+0000","last_clean":"2024-03-16T20:33:55.111308+0000","last_became_active":"2024-03-16T20:33:49.855957+0000","last_became_peered":"2024-03-16T20:33:49.855957+0000","last_unstale":"2024-03-16T20:33:55.111308+0000","last_undegraded":"2024-03-16T20:33:55.111308+0000","last_fullsized":"2024-03-16T20:33:55.111308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:07.035334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.112291+0000","last_change":"2024-03-16T20:33:53.892149+0000","last_active":"2024-03-16T20:33:55.112291+0000","last_peered":"2024-03-16T20:33:55.112291+0000","last_clean":"2024-03-16T20:33:55.112291+0000","last_became_active":"2024-03-16T20:33:53.891730+0000","last_became_peered":"2024-03-16T20:33:53.891730+0000","last_unstale":"2024-03-16T20:33:55.112291+0000","last_undegraded":"2024-03-16T20:33:55.112291+0000","last_fullsized":"2024-03-16T20:33:55.112291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:59.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.926374+0000","last_change":"2024-03-16T20:33:48.849909+0000","last_active":"2024-03-16T20:34:08.926374+0000","last_peered":"2024-03-16T20:34:08.926374+0000","last_clean":"2024-03-16T20:34:08.926374+0000","last_became_active":"2024-03-16T20:33:48.848645+0000","last_became_peered":"2024-03-16T20:33:48.848645+0000","last_unstale":"2024-03-16T20:34:08.926374+0000","last_undegraded":"2024-03-16T20:34:08.926374+0000","last_fullsized":"2024-03-16T20:34:08.926374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:44.401679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871609+0000","last_change":"2024-03-16T20:33:49.853100+0000","last_active":"2024-03-16T20:33:54.871609+0000","last_peered":"2024-03-16T20:33:54.871609+0000","last_clean":"2024-03-16T20:33:54.871609+0000","last_became_active":"2024-03-16T20:33:49.852743+0000","last_became_peered":"2024-03-16T20:33:49.852743+0000","last_unstale":"2024-03-16T20:33:54.871609+0000","last_undegraded":"2024-03-16T20:33:54.871609+0000","last_fullsized":"2024-03-16T20:33:54.871609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:42.664026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110319+0000","last_change":"2024-03-16T20:33:53.881626+0000","last_active":"2024-03-16T20:33:55.110319+0000","last_peered":"2024-03-16T20:33:55.110319+0000","last_clean":"2024-03-16T20:33:55.110319+0000","last_became_active":"2024-03-16T20:33:53.881205+0000","last_became_peered":"2024-03-16T20:33:53.881205+0000","last_unstale":"2024-03-16T20:33:55.110319+0000","last_undegraded":"2024-03-16T20:33:55.110319+0000","last_fullsized":"2024-03-16T20:33:55.110319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:19.525953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'21","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:11.355167+0000","last_change":"2024-03-16T20:33:48.848490+0000","last_active":"2024-03-16T20:34:11.355167+0000","last_peered":"2024-03-16T20:34:11.355167+0000","last_clean":"2024-03-16T20:34:11.355167+0000","last_became_active":"2024-03-16T20:33:48.848152+0000","last_became_peered":"2024-03-16T20:33:48.848152+0000","last_unstale":"2024-03-16T20:34:11.355167+0000","last_undegraded":"2024-03-16T20:34:11.355167+0000","last_fullsized":"2024-03-16T20:34:11.355167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:43:19.908178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":23,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110228+0000","last_change":"2024-03-16T20:33:49.851269+0000","last_active":"2024-03-16T20:33:55.110228+0000","last_peered":"2024-03-16T20:33:55.110228+0000","last_clean":"2024-03-16T20:33:55.110228+0000","last_became_active":"2024-03-16T20:33:49.850963+0000","last_became_peered":"2024-03-16T20:33:49.850963+0000","last_unstale":"2024-03-16T20:33:55.110228+0000","last_undegraded":"2024-03-16T20:33:55.110228+0000","last_fullsized":"2024-03-16T20:33:55.110228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:26:01.312043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.875099+0000","last_change":"2024-03-16T20:33:53.889419+0000","last_active":"2024-03-16T20:33:54.875099+0000","last_peered":"2024-03-16T20:33:54.875099+0000","last_clean":"2024-03-16T20:33:54.875099+0000","last_became_active":"2024-03-16T20:33:53.889016+0000","last_became_peered":"2024-03-16T20:33:53.889016+0000","last_unstale":"2024-03-16T20:33:54.875099+0000","last_undegraded":"2024-03-16T20:33:54.875099+0000","last_fullsized":"2024-03-16T20:33:54.875099+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:32:58.333441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940274+0000","last_change":"2024-03-16T20:33:48.855070+0000","last_active":"2024-03-16T20:34:09.940274+0000","last_peered":"2024-03-16T20:34:09.940274+0000","last_clean":"2024-03-16T20:34:09.940274+0000","last_became_active":"2024-03-16T20:33:48.854669+0000","last_became_peered":"2024-03-16T20:33:48.854669+0000","last_unstale":"2024-03-16T20:34:09.940274+0000","last_undegraded":"2024-03-16T20:34:09.940274+0000","last_fullsized":"2024-03-16T20:34:09.940274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:38.229641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873073+0000","last_change":"2024-03-16T20:33:49.850186+0000","last_active":"2024-03-16T20:33:54.873073+0000","last_peered":"2024-03-16T20:33:54.873073+0000","last_clean":"2024-03-16T20:33:54.873073+0000","last_became_active":"2024-03-16T20:33:49.849811+0000","last_became_peered":"2024-03-16T20:33:49.849811+0000","last_unstale":"2024-03-16T20:33:54.873073+0000","last_undegraded":"2024-03-16T20:33:54.873073+0000","last_fullsized":"2024-03-16T20:33:54.873073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:13:03.399984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156428+0000","last_change":"2024-03-16T20:33:53.890030+0000","last_active":"2024-03-16T20:33:55.156428+0000","last_peered":"2024-03-16T20:33:55.156428+0000","last_clean":"2024-03-16T20:33:55.156428+0000","last_became_active":"2024-03-16T20:33:53.889602+0000","last_became_peered":"2024-03-16T20:33:53.889602+0000","last_unstale":"2024-03-16T20:33:55.156428+0000","last_undegraded":"2024-03-16T20:33:55.156428+0000","last_fullsized":"2024-03-16T20:33:55.156428+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:04.319150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.711864+0000","last_change":"2024-03-16T20:33:48.840422+0000","last_active":"2024-03-16T20:34:07.711864+0000","last_peered":"2024-03-16T20:34:07.711864+0000","last_clean":"2024-03-16T20:34:07.711864+0000","last_became_active":"2024-03-16T20:33:48.840125+0000","last_became_peered":"2024-03-16T20:33:48.840125+0000","last_unstale":"2024-03-16T20:34:07.711864+0000","last_undegraded":"2024-03-16T20:34:07.711864+0000","last_fullsized":"2024-03-16T20:34:07.711864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:05.397252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110638+0000","last_change":"2024-03-16T20:33:49.862475+0000","last_active":"2024-03-16T20:33:55.110638+0000","last_peered":"2024-03-16T20:33:55.110638+0000","last_clean":"2024-03-16T20:33:55.110638+0000","last_became_active":"2024-03-16T20:33:49.862032+0000","last_became_peered":"2024-03-16T20:33:49.862032+0000","last_unstale":"2024-03-16T20:33:55.110638+0000","last_undegraded":"2024-03-16T20:33:55.110638+0000","last_fullsized":"2024-03-16T20:33:55.110638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:16:53.319352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110920+0000","last_change":"2024-03-16T20:33:53.884019+0000","last_active":"2024-03-16T20:33:55.110920+0000","last_peered":"2024-03-16T20:33:55.110920+0000","last_clean":"2024-03-16T20:33:55.110920+0000","last_became_active":"2024-03-16T20:33:53.883608+0000","last_became_peered":"2024-03-16T20:33:53.883608+0000","last_unstale":"2024-03-16T20:33:55.110920+0000","last_undegraded":"2024-03-16T20:33:55.110920+0000","last_fullsized":"2024-03-16T20:33:55.110920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:58.689682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.932185+0000","last_change":"2024-03-16T20:33:48.848145+0000","last_active":"2024-03-16T20:34:08.932185+0000","last_peered":"2024-03-16T20:34:08.932185+0000","last_clean":"2024-03-16T20:34:08.932185+0000","last_became_active":"2024-03-16T20:33:48.847754+0000","last_became_peered":"2024-03-16T20:33:48.847754+0000","last_unstale":"2024-03-16T20:34:08.932185+0000","last_undegraded":"2024-03-16T20:34:08.932185+0000","last_fullsized":"2024-03-16T20:34:08.932185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:59.967346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110207+0000","last_change":"2024-03-16T20:33:49.851307+0000","last_active":"2024-03-16T20:33:55.110207+0000","last_peered":"2024-03-16T20:33:55.110207+0000","last_clean":"2024-03-16T20:33:55.110207+0000","last_became_active":"2024-03-16T20:33:49.850930+0000","last_became_peered":"2024-03-16T20:33:49.850930+0000","last_unstale":"2024-03-16T20:33:55.110207+0000","last_undegraded":"2024-03-16T20:33:55.110207+0000","last_fullsized":"2024-03-16T20:33:55.110207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:23:54.528895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873710+0000","last_change":"2024-03-16T20:33:53.893149+0000","last_active":"2024-03-16T20:33:54.873710+0000","last_peered":"2024-03-16T20:33:54.873710+0000","last_clean":"2024-03-16T20:33:54.873710+0000","last_became_active":"2024-03-16T20:33:53.892532+0000","last_became_peered":"2024-03-16T20:33:53.892532+0000","last_unstale":"2024-03-16T20:33:54.873710+0000","last_undegraded":"2024-03-16T20:33:54.873710+0000","last_fullsized":"2024-03-16T20:33:54.873710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:09.408974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.935261+0000","last_change":"2024-03-16T20:33:48.849788+0000","last_active":"2024-03-16T20:34:08.935261+0000","last_peered":"2024-03-16T20:34:08.935261+0000","last_clean":"2024-03-16T20:34:08.935261+0000","last_became_active":"2024-03-16T20:33:48.849360+0000","last_became_peered":"2024-03-16T20:33:48.849360+0000","last_unstale":"2024-03-16T20:34:08.935261+0000","last_undegraded":"2024-03-16T20:34:08.935261+0000","last_fullsized":"2024-03-16T20:34:08.935261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:29.026216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874438+0000","last_change":"2024-03-16T20:33:49.856702+0000","last_active":"2024-03-16T20:33:54.874438+0000","last_peered":"2024-03-16T20:33:54.874438+0000","last_clean":"2024-03-16T20:33:54.874438+0000","last_became_active":"2024-03-16T20:33:49.856373+0000","last_became_peered":"2024-03-16T20:33:49.856373+0000","last_unstale":"2024-03-16T20:33:54.874438+0000","last_undegraded":"2024-03-16T20:33:54.874438+0000","last_fullsized":"2024-03-16T20:33:54.874438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:31:40.093440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874500+0000","last_change":"2024-03-16T20:33:53.884193+0000","last_active":"2024-03-16T20:33:54.874500+0000","last_peered":"2024-03-16T20:33:54.874500+0000","last_clean":"2024-03-16T20:33:54.874500+0000","last_became_active":"2024-03-16T20:33:53.883763+0000","last_became_peered":"2024-03-16T20:33:53.883763+0000","last_unstale":"2024-03-16T20:33:54.874500+0000","last_undegraded":"2024-03-16T20:33:54.874500+0000","last_fullsized":"2024-03-16T20:33:54.874500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:50:20.969662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.923969+0000","last_change":"2024-03-16T20:33:48.851166+0000","last_active":"2024-03-16T20:34:08.923969+0000","last_peered":"2024-03-16T20:34:08.923969+0000","last_clean":"2024-03-16T20:34:08.923969+0000","last_became_active":"2024-03-16T20:33:48.850843+0000","last_became_peered":"2024-03-16T20:33:48.850843+0000","last_unstale":"2024-03-16T20:34:08.923969+0000","last_undegraded":"2024-03-16T20:34:08.923969+0000","last_fullsized":"2024-03-16T20:34:08.923969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:09.142942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156599+0000","last_change":"2024-03-16T20:33:49.860038+0000","last_active":"2024-03-16T20:33:55.156599+0000","last_peered":"2024-03-16T20:33:55.156599+0000","last_clean":"2024-03-16T20:33:55.156599+0000","last_became_active":"2024-03-16T20:33:49.859715+0000","last_became_peered":"2024-03-16T20:33:49.859715+0000","last_unstale":"2024-03-16T20:33:55.156599+0000","last_undegraded":"2024-03-16T20:33:55.156599+0000","last_fullsized":"2024-03-16T20:33:55.156599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:27:32.800166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109990+0000","last_change":"2024-03-16T20:33:53.930129+0000","last_active":"2024-03-16T20:33:55.109990+0000","last_peered":"2024-03-16T20:33:55.109990+0000","last_clean":"2024-03-16T20:33:55.109990+0000","last_became_active":"2024-03-16T20:33:53.929655+0000","last_became_peered":"2024-03-16T20:33:53.929655+0000","last_unstale":"2024-03-16T20:33:55.109990+0000","last_undegraded":"2024-03-16T20:33:55.109990+0000","last_fullsized":"2024-03-16T20:33:55.109990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:46.422720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110623+0000","last_change":"2024-03-16T20:33:48.849089+0000","last_active":"2024-03-16T20:33:55.110623+0000","last_peered":"2024-03-16T20:33:55.110623+0000","last_clean":"2024-03-16T20:33:55.110623+0000","last_became_active":"2024-03-16T20:33:48.848719+0000","last_became_peered":"2024-03-16T20:33:48.848719+0000","last_unstale":"2024-03-16T20:33:55.110623+0000","last_undegraded":"2024-03-16T20:33:55.110623+0000","last_fullsized":"2024-03-16T20:33:55.110623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:31:23.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111549+0000","last_change":"2024-03-16T20:33:49.853247+0000","last_active":"2024-03-16T20:33:55.111549+0000","last_peered":"2024-03-16T20:33:55.111549+0000","last_clean":"2024-03-16T20:33:55.111549+0000","last_became_active":"2024-03-16T20:33:49.852870+0000","last_became_peered":"2024-03-16T20:33:49.852870+0000","last_unstale":"2024-03-16T20:33:55.111549+0000","last_undegraded":"2024-03-16T20:33:55.111549+0000","last_fullsized":"2024-03-16T20:33:55.111549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:56.365484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873655+0000","last_change":"2024-03-16T20:33:53.887350+0000","last_active":"2024-03-16T20:33:54.873655+0000","last_peered":"2024-03-16T20:33:54.873655+0000","last_clean":"2024-03-16T20:33:54.873655+0000","last_became_active":"2024-03-16T20:33:53.886782+0000","last_became_peered":"2024-03-16T20:33:53.886782+0000","last_unstale":"2024-03-16T20:33:54.873655+0000","last_undegraded":"2024-03-16T20:33:54.873655+0000","last_fullsized":"2024-03-16T20:33:54.873655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:00:39.898430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110702+0000","last_change":"2024-03-16T20:33:48.851543+0000","last_active":"2024-03-16T20:33:55.110702+0000","last_peered":"2024-03-16T20:33:55.110702+0000","last_clean":"2024-03-16T20:33:55.110702+0000","last_became_active":"2024-03-16T20:33:48.851196+0000","last_became_peered":"2024-03-16T20:33:48.851196+0000","last_unstale":"2024-03-16T20:33:55.110702+0000","last_undegraded":"2024-03-16T20:33:55.110702+0000","last_fullsized":"2024-03-16T20:33:55.110702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:06.309259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111826+0000","last_change":"2024-03-16T20:33:49.856021+0000","last_active":"2024-03-16T20:33:55.111826+0000","last_peered":"2024-03-16T20:33:55.111826+0000","last_clean":"2024-03-16T20:33:55.111826+0000","last_became_active":"2024-03-16T20:33:49.854130+0000","last_became_peered":"2024-03-16T20:33:49.854130+0000","last_unstale":"2024-03-16T20:33:55.111826+0000","last_undegraded":"2024-03-16T20:33:55.111826+0000","last_fullsized":"2024-03-16T20:33:55.111826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:29:39.970391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873619+0000","last_change":"2024-03-16T20:33:53.930094+0000","last_active":"2024-03-16T20:33:54.873619+0000","last_peered":"2024-03-16T20:33:54.873619+0000","last_clean":"2024-03-16T20:33:54.873619+0000","last_became_active":"2024-03-16T20:33:53.929613+0000","last_became_peered":"2024-03-16T20:33:53.929613+0000","last_unstale":"2024-03-16T20:33:54.873619+0000","last_undegraded":"2024-03-16T20:33:54.873619+0000","last_fullsized":"2024-03-16T20:33:54.873619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:30.680295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874565+0000","last_change":"2024-03-16T20:33:49.857949+0000","last_active":"2024-03-16T20:33:54.874565+0000","last_peered":"2024-03-16T20:33:54.874565+0000","last_clean":"2024-03-16T20:33:54.874565+0000","last_became_active":"2024-03-16T20:33:49.857642+0000","last_became_peered":"2024-03-16T20:33:49.857642+0000","last_unstale":"2024-03-16T20:33:54.874565+0000","last_undegraded":"2024-03-16T20:33:54.874565+0000","last_fullsized":"2024-03-16T20:33:54.874565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:23.040787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.928996+0000","last_change":"2024-03-16T20:33:48.856908+0000","last_active":"2024-03-16T20:34:09.928996+0000","last_peered":"2024-03-16T20:34:09.928996+0000","last_clean":"2024-03-16T20:34:09.928996+0000","last_became_active":"2024-03-16T20:33:48.856578+0000","last_became_peered":"2024-03-16T20:33:48.856578+0000","last_unstale":"2024-03-16T20:34:09.928996+0000","last_undegraded":"2024-03-16T20:34:09.928996+0000","last_fullsized":"2024-03-16T20:34:09.928996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:37.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111025+0000","last_change":"2024-03-16T20:33:53.881965+0000","last_active":"2024-03-16T20:33:55.111025+0000","last_peered":"2024-03-16T20:33:55.111025+0000","last_clean":"2024-03-16T20:33:55.111025+0000","last_became_active":"2024-03-16T20:33:53.881247+0000","last_became_peered":"2024-03-16T20:33:53.881247+0000","last_unstale":"2024-03-16T20:33:55.111025+0000","last_undegraded":"2024-03-16T20:33:55.111025+0000","last_fullsized":"2024-03-16T20:33:55.111025+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:18.868143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109858+0000","last_change":"2024-03-16T20:33:49.857605+0000","last_active":"2024-03-16T20:33:55.109858+0000","last_peered":"2024-03-16T20:33:55.109858+0000","last_clean":"2024-03-16T20:33:55.109858+0000","last_became_active":"2024-03-16T20:33:49.857242+0000","last_became_peered":"2024-03-16T20:33:49.857242+0000","last_unstale":"2024-03-16T20:33:55.109858+0000","last_undegraded":"2024-03-16T20:33:55.109858+0000","last_fullsized":"2024-03-16T20:33:55.109858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:55:21.442023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.933731+0000","last_change":"2024-03-16T20:33:48.850177+0000","last_active":"2024-03-16T20:34:08.933731+0000","last_peered":"2024-03-16T20:34:08.933731+0000","last_clean":"2024-03-16T20:34:08.933731+0000","last_became_active":"2024-03-16T20:33:48.849838+0000","last_became_peered":"2024-03-16T20:33:48.849838+0000","last_unstale":"2024-03-16T20:34:08.933731+0000","last_undegraded":"2024-03-16T20:34:08.933731+0000","last_fullsized":"2024-03-16T20:34:08.933731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:24.544880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110076+0000","last_change":"2024-03-16T20:32:34.913833+0000","last_active":"2024-03-16T20:33:55.110076+0000","last_peered":"2024-03-16T20:33:55.110076+0000","last_clean":"2024-03-16T20:33:55.110076+0000","last_became_active":"2024-03-16T20:32:34.913443+0000","last_became_peered":"2024-03-16T20:32:34.913443+0000","last_unstale":"2024-03-16T20:33:55.110076+0000","last_undegraded":"2024-03-16T20:33:55.110076+0000","last_fullsized":"2024-03-16T20:33:55.110076+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_clean_scrub_stamp":"2024-03-16T20:29:10.760383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:05.333386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874820+0000","last_change":"2024-03-16T20:33:53.931311+0000","last_active":"2024-03-16T20:33:54.874820+0000","last_peered":"2024-03-16T20:33:54.874820+0000","last_clean":"2024-03-16T20:33:54.874820+0000","last_became_active":"2024-03-16T20:33:53.930972+0000","last_became_peered":"2024-03-16T20:33:53.930972+0000","last_unstale":"2024-03-16T20:33:54.874820+0000","last_undegraded":"2024-03-16T20:33:54.874820+0000","last_fullsized":"2024-03-16T20:33:54.874820+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:29.776651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.872757+0000","last_change":"2024-03-16T20:33:49.838866+0000","last_active":"2024-03-16T20:33:54.872757+0000","last_peered":"2024-03-16T20:33:54.872757+0000","last_clean":"2024-03-16T20:33:54.872757+0000","last_became_active":"2024-03-16T20:33:49.838561+0000","last_became_peered":"2024-03-16T20:33:49.838561+0000","last_unstale":"2024-03-16T20:33:54.872757+0000","last_undegraded":"2024-03-16T20:33:54.872757+0000","last_fullsized":"2024-03-16T20:33:54.872757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:57:21.098078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.724982+0000","last_change":"2024-03-16T20:33:48.847098+0000","last_active":"2024-03-16T20:34:08.724982+0000","last_peered":"2024-03-16T20:34:08.724982+0000","last_clean":"2024-03-16T20:34:08.724982+0000","last_became_active":"2024-03-16T20:33:48.846399+0000","last_became_peered":"2024-03-16T20:33:48.846399+0000","last_unstale":"2024-03-16T20:34:08.724982+0000","last_undegraded":"2024-03-16T20:34:08.724982+0000","last_fullsized":"2024-03-16T20:34:08.724982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:59.126547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156422+0000","last_change":"2024-03-16T20:33:53.890952+0000","last_active":"2024-03-16T20:33:55.156422+0000","last_peered":"2024-03-16T20:33:55.156422+0000","last_clean":"2024-03-16T20:33:55.156422+0000","last_became_active":"2024-03-16T20:33:53.890473+0000","last_became_peered":"2024-03-16T20:33:53.890473+0000","last_unstale":"2024-03-16T20:33:55.156422+0000","last_undegraded":"2024-03-16T20:33:55.156422+0000","last_fullsized":"2024-03-16T20:33:55.156422+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:06.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.935486+0000","last_change":"2024-03-16T20:33:48.851220+0000","last_active":"2024-03-16T20:34:08.935486+0000","last_peered":"2024-03-16T20:34:08.935486+0000","last_clean":"2024-03-16T20:34:08.935486+0000","last_became_active":"2024-03-16T20:33:48.850927+0000","last_became_peered":"2024-03-16T20:33:48.850927+0000","last_unstale":"2024-03-16T20:34:08.935486+0000","last_undegraded":"2024-03-16T20:34:08.935486+0000","last_fullsized":"2024-03-16T20:34:08.935486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:50.105276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873629+0000","last_change":"2024-03-16T20:33:49.850272+0000","last_active":"2024-03-16T20:33:54.873629+0000","last_peered":"2024-03-16T20:33:54.873629+0000","last_clean":"2024-03-16T20:33:54.873629+0000","last_became_active":"2024-03-16T20:33:49.849913+0000","last_became_peered":"2024-03-16T20:33:49.849913+0000","last_unstale":"2024-03-16T20:33:54.873629+0000","last_undegraded":"2024-03-16T20:33:54.873629+0000","last_fullsized":"2024-03-16T20:33:54.873629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:53:18.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110943+0000","last_change":"2024-03-16T20:33:53.930421+0000","last_active":"2024-03-16T20:33:55.110943+0000","last_peered":"2024-03-16T20:33:55.110943+0000","last_clean":"2024-03-16T20:33:55.110943+0000","last_became_active":"2024-03-16T20:33:53.929957+0000","last_became_peered":"2024-03-16T20:33:53.929957+0000","last_unstale":"2024-03-16T20:33:55.110943+0000","last_undegraded":"2024-03-16T20:33:55.110943+0000","last_fullsized":"2024-03-16T20:33:55.110943+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:20.541592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.936089+0000","last_change":"2024-03-16T20:33:48.845096+0000","last_active":"2024-03-16T20:34:08.936089+0000","last_peered":"2024-03-16T20:34:08.936089+0000","last_clean":"2024-03-16T20:34:08.936089+0000","last_became_active":"2024-03-16T20:33:48.844580+0000","last_became_peered":"2024-03-16T20:33:48.844580+0000","last_unstale":"2024-03-16T20:34:08.936089+0000","last_undegraded":"2024-03-16T20:34:08.936089+0000","last_fullsized":"2024-03-16T20:34:08.936089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:56:53.857336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109948+0000","last_change":"2024-03-16T20:33:49.858488+0000","last_active":"2024-03-16T20:33:55.109948+0000","last_peered":"2024-03-16T20:33:55.109948+0000","last_clean":"2024-03-16T20:33:55.109948+0000","last_became_active":"2024-03-16T20:33:49.858163+0000","last_became_peered":"2024-03-16T20:33:49.858163+0000","last_unstale":"2024-03-16T20:33:55.109948+0000","last_undegraded":"2024-03-16T20:33:55.109948+0000","last_fullsized":"2024-03-16T20:33:55.109948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:49.399329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110376+0000","last_change":"2024-03-16T20:33:53.884240+0000","last_active":"2024-03-16T20:33:55.110376+0000","last_peered":"2024-03-16T20:33:55.110376+0000","last_clean":"2024-03-16T20:33:55.110376+0000","last_became_active":"2024-03-16T20:33:53.883864+0000","last_became_peered":"2024-03-16T20:33:53.883864+0000","last_unstale":"2024-03-16T20:33:55.110376+0000","last_undegraded":"2024-03-16T20:33:55.110376+0000","last_fullsized":"2024-03-16T20:33:55.110376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:57:56.073112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940664+0000","last_change":"2024-03-16T20:33:48.858217+0000","last_active":"2024-03-16T20:34:09.940664+0000","last_peered":"2024-03-16T20:34:09.940664+0000","last_clean":"2024-03-16T20:34:09.940664+0000","last_became_active":"2024-03-16T20:33:48.857844+0000","last_became_peered":"2024-03-16T20:33:48.857844+0000","last_unstale":"2024-03-16T20:34:09.940664+0000","last_undegraded":"2024-03-16T20:34:09.940664+0000","last_fullsized":"2024-03-16T20:34:09.940664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:09.143250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110824+0000","last_change":"2024-03-16T20:33:49.856002+0000","last_active":"2024-03-16T20:33:55.110824+0000","last_peered":"2024-03-16T20:33:55.110824+0000","last_clean":"2024-03-16T20:33:55.110824+0000","last_became_active":"2024-03-16T20:33:49.855664+0000","last_became_peered":"2024-03-16T20:33:49.855664+0000","last_unstale":"2024-03-16T20:33:55.110824+0000","last_undegraded":"2024-03-16T20:33:55.110824+0000","last_fullsized":"2024-03-16T20:33:55.110824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:55:36.502729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111392+0000","last_change":"2024-03-16T20:33:53.931291+0000","last_active":"2024-03-16T20:33:55.111392+0000","last_peered":"2024-03-16T20:33:55.111392+0000","last_clean":"2024-03-16T20:33:55.111392+0000","last_became_active":"2024-03-16T20:33:53.930863+0000","last_became_peered":"2024-03-16T20:33:53.930863+0000","last_unstale":"2024-03-16T20:33:55.111392+0000","last_undegraded":"2024-03-16T20:33:55.111392+0000","last_fullsized":"2024-03-16T20:33:55.111392+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:36:59.977054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.711835+0000","last_change":"2024-03-16T20:33:48.855853+0000","last_active":"2024-03-16T20:34:07.711835+0000","last_peered":"2024-03-16T20:34:07.711835+0000","last_clean":"2024-03-16T20:34:07.711835+0000","last_became_active":"2024-03-16T20:33:48.855539+0000","last_became_peered":"2024-03-16T20:33:48.855539+0000","last_unstale":"2024-03-16T20:34:07.711835+0000","last_undegraded":"2024-03-16T20:34:07.711835+0000","last_fullsized":"2024-03-16T20:34:07.711835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:56:48.618350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109638+0000","last_change":"2024-03-16T20:33:49.854825+0000","last_active":"2024-03-16T20:33:55.109638+0000","last_peered":"2024-03-16T20:33:55.109638+0000","last_clean":"2024-03-16T20:33:55.109638+0000","last_became_active":"2024-03-16T20:33:49.854396+0000","last_became_peered":"2024-03-16T20:33:49.854396+0000","last_unstale":"2024-03-16T20:33:55.109638+0000","last_undegraded":"2024-03-16T20:33:55.109638+0000","last_fullsized":"2024-03-16T20:33:55.109638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:31:18.133694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111224+0000","last_change":"2024-03-16T20:33:53.881554+0000","last_active":"2024-03-16T20:33:55.111224+0000","last_peered":"2024-03-16T20:33:55.111224+0000","last_clean":"2024-03-16T20:33:55.111224+0000","last_became_active":"2024-03-16T20:33:53.881030+0000","last_became_peered":"2024-03-16T20:33:53.881030+0000","last_unstale":"2024-03-16T20:33:55.111224+0000","last_undegraded":"2024-03-16T20:33:55.111224+0000","last_fullsized":"2024-03-16T20:33:55.111224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:46.499466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.940896+0000","last_change":"2024-03-16T20:33:48.853138+0000","last_active":"2024-03-16T20:34:09.940896+0000","last_peered":"2024-03-16T20:34:09.940896+0000","last_clean":"2024-03-16T20:34:09.940896+0000","last_became_active":"2024-03-16T20:33:48.851501+0000","last_became_peered":"2024-03-16T20:33:48.851501+0000","last_unstale":"2024-03-16T20:34:09.940896+0000","last_undegraded":"2024-03-16T20:34:09.940896+0000","last_fullsized":"2024-03-16T20:34:09.940896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:57:40.368614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156250+0000","last_change":"2024-03-16T20:33:49.850307+0000","last_active":"2024-03-16T20:33:55.156250+0000","last_peered":"2024-03-16T20:33:55.156250+0000","last_clean":"2024-03-16T20:33:55.156250+0000","last_became_active":"2024-03-16T20:33:49.849953+0000","last_became_peered":"2024-03-16T20:33:49.849953+0000","last_unstale":"2024-03-16T20:33:55.156250+0000","last_undegraded":"2024-03-16T20:33:55.156250+0000","last_fullsized":"2024-03-16T20:33:55.156250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:36:51.020011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111483+0000","last_change":"2024-03-16T20:33:53.892370+0000","last_active":"2024-03-16T20:33:55.111483+0000","last_peered":"2024-03-16T20:33:55.111483+0000","last_clean":"2024-03-16T20:33:55.111483+0000","last_became_active":"2024-03-16T20:33:53.891885+0000","last_became_peered":"2024-03-16T20:33:53.891885+0000","last_unstale":"2024-03-16T20:33:55.111483+0000","last_undegraded":"2024-03-16T20:33:55.111483+0000","last_fullsized":"2024-03-16T20:33:55.111483+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:30.770882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.707308+0000","last_change":"2024-03-16T20:33:48.855295+0000","last_active":"2024-03-16T20:34:09.707308+0000","last_peered":"2024-03-16T20:34:09.707308+0000","last_clean":"2024-03-16T20:34:09.707308+0000","last_became_active":"2024-03-16T20:33:48.854970+0000","last_became_peered":"2024-03-16T20:33:48.854970+0000","last_unstale":"2024-03-16T20:34:09.707308+0000","last_undegraded":"2024-03-16T20:34:09.707308+0000","last_fullsized":"2024-03-16T20:34:09.707308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:20:28.281615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110410+0000","last_change":"2024-03-16T20:33:49.861637+0000","last_active":"2024-03-16T20:33:55.110410+0000","last_peered":"2024-03-16T20:33:55.110410+0000","last_clean":"2024-03-16T20:33:55.110410+0000","last_became_active":"2024-03-16T20:33:49.861284+0000","last_became_peered":"2024-03-16T20:33:49.861284+0000","last_unstale":"2024-03-16T20:33:55.110410+0000","last_undegraded":"2024-03-16T20:33:55.110410+0000","last_fullsized":"2024-03-16T20:33:55.110410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.937928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111442+0000","last_change":"2024-03-16T20:33:53.931633+0000","last_active":"2024-03-16T20:33:55.111442+0000","last_peered":"2024-03-16T20:33:55.111442+0000","last_clean":"2024-03-16T20:33:55.111442+0000","last_became_active":"2024-03-16T20:33:53.930633+0000","last_became_peered":"2024-03-16T20:33:53.930633+0000","last_unstale":"2024-03-16T20:33:55.111442+0000","last_undegraded":"2024-03-16T20:33:55.111442+0000","last_fullsized":"2024-03-16T20:33:55.111442+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:33:57.356317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:07.712199+0000","last_change":"2024-03-16T20:33:48.844925+0000","last_active":"2024-03-16T20:34:07.712199+0000","last_peered":"2024-03-16T20:34:07.712199+0000","last_clean":"2024-03-16T20:34:07.712199+0000","last_became_active":"2024-03-16T20:33:48.844569+0000","last_became_peered":"2024-03-16T20:33:48.844569+0000","last_unstale":"2024-03-16T20:34:07.712199+0000","last_undegraded":"2024-03-16T20:34:07.712199+0000","last_fullsized":"2024-03-16T20:34:07.712199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:04.430554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110846+0000","last_change":"2024-03-16T20:33:49.859085+0000","last_active":"2024-03-16T20:33:55.110846+0000","last_peered":"2024-03-16T20:33:55.110846+0000","last_clean":"2024-03-16T20:33:55.110846+0000","last_became_active":"2024-03-16T20:33:49.858637+0000","last_became_peered":"2024-03-16T20:33:49.858637+0000","last_unstale":"2024-03-16T20:33:55.110846+0000","last_undegraded":"2024-03-16T20:33:55.110846+0000","last_fullsized":"2024-03-16T20:33:55.110846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:22:17.020669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871355+0000","last_change":"2024-03-16T20:33:53.891866+0000","last_active":"2024-03-16T20:33:54.871355+0000","last_peered":"2024-03-16T20:33:54.871355+0000","last_clean":"2024-03-16T20:33:54.871355+0000","last_became_active":"2024-03-16T20:33:53.891424+0000","last_became_peered":"2024-03-16T20:33:53.891424+0000","last_unstale":"2024-03-16T20:33:54.871355+0000","last_undegraded":"2024-03-16T20:33:54.871355+0000","last_fullsized":"2024-03-16T20:33:54.871355+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.932053+0000","last_change":"2024-03-16T20:33:48.856124+0000","last_active":"2024-03-16T20:34:08.932053+0000","last_peered":"2024-03-16T20:34:08.932053+0000","last_clean":"2024-03-16T20:34:08.932053+0000","last_became_active":"2024-03-16T20:33:48.855813+0000","last_became_peered":"2024-03-16T20:33:48.855813+0000","last_unstale":"2024-03-16T20:34:08.932053+0000","last_undegraded":"2024-03-16T20:34:08.932053+0000","last_fullsized":"2024-03-16T20:34:08.932053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:55.122457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111209+0000","last_change":"2024-03-16T20:33:49.857479+0000","last_active":"2024-03-16T20:33:55.111209+0000","last_peered":"2024-03-16T20:33:55.111209+0000","last_clean":"2024-03-16T20:33:55.111209+0000","last_became_active":"2024-03-16T20:33:49.856121+0000","last_became_peered":"2024-03-16T20:33:49.856121+0000","last_unstale":"2024-03-16T20:33:55.111209+0000","last_undegraded":"2024-03-16T20:33:55.111209+0000","last_fullsized":"2024-03-16T20:33:55.111209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:08:55.613977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110613+0000","last_change":"2024-03-16T20:33:53.886802+0000","last_active":"2024-03-16T20:33:55.110613+0000","last_peered":"2024-03-16T20:33:55.110613+0000","last_clean":"2024-03-16T20:33:55.110613+0000","last_became_active":"2024-03-16T20:33:53.886430+0000","last_became_peered":"2024-03-16T20:33:53.886430+0000","last_unstale":"2024-03-16T20:33:55.110613+0000","last_undegraded":"2024-03-16T20:33:55.110613+0000","last_fullsized":"2024-03-16T20:33:55.110613+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:45.673571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:08.923823+0000","last_change":"2024-03-16T20:33:48.846628+0000","last_active":"2024-03-16T20:34:08.923823+0000","last_peered":"2024-03-16T20:34:08.923823+0000","last_clean":"2024-03-16T20:34:08.923823+0000","last_became_active":"2024-03-16T20:33:48.846276+0000","last_became_peered":"2024-03-16T20:33:48.846276+0000","last_unstale":"2024-03-16T20:34:08.923823+0000","last_undegraded":"2024-03-16T20:34:08.923823+0000","last_fullsized":"2024-03-16T20:34:08.923823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:24.323938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110768+0000","last_change":"2024-03-16T20:33:49.854041+0000","last_active":"2024-03-16T20:33:55.110768+0000","last_peered":"2024-03-16T20:33:55.110768+0000","last_clean":"2024-03-16T20:33:55.110768+0000","last_became_active":"2024-03-16T20:33:49.853653+0000","last_became_peered":"2024-03-16T20:33:49.853653+0000","last_unstale":"2024-03-16T20:33:55.110768+0000","last_undegraded":"2024-03-16T20:33:55.110768+0000","last_fullsized":"2024-03-16T20:33:55.110768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:59:10.498314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874298+0000","last_change":"2024-03-16T20:33:53.884292+0000","last_active":"2024-03-16T20:33:54.874298+0000","last_peered":"2024-03-16T20:33:54.874298+0000","last_clean":"2024-03-16T20:33:54.874298+0000","last_became_active":"2024-03-16T20:33:53.883778+0000","last_became_peered":"2024-03-16T20:33:53.883778+0000","last_unstale":"2024-03-16T20:33:54.874298+0000","last_undegraded":"2024-03-16T20:33:54.874298+0000","last_fullsized":"2024-03-16T20:33:54.874298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:33.877142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:34:09.941110+0000","last_change":"2024-03-16T20:33:48.845277+0000","last_active":"2024-03-16T20:34:09.941110+0000","last_peered":"2024-03-16T20:34:09.941110+0000","last_clean":"2024-03-16T20:34:09.941110+0000","last_became_active":"2024-03-16T20:33:48.844954+0000","last_became_peered":"2024-03-16T20:33:48.844954+0000","last_unstale":"2024-03-16T20:34:09.941110+0000","last_undegraded":"2024-03-16T20:34:09.941110+0000","last_fullsized":"2024-03-16T20:34:09.941110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:24:59.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.111140+0000","last_change":"2024-03-16T20:33:49.856152+0000","last_active":"2024-03-16T20:33:55.111140+0000","last_peered":"2024-03-16T20:33:55.111140+0000","last_clean":"2024-03-16T20:33:55.111140+0000","last_became_active":"2024-03-16T20:33:49.855026+0000","last_became_peered":"2024-03-16T20:33:49.855026+0000","last_unstale":"2024-03-16T20:33:55.111140+0000","last_undegraded":"2024-03-16T20:33:55.111140+0000","last_fullsized":"2024-03-16T20:33:55.111140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:40.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156309+0000","last_change":"2024-03-16T20:33:49.851539+0000","last_active":"2024-03-16T20:33:55.156309+0000","last_peered":"2024-03-16T20:33:55.156309+0000","last_clean":"2024-03-16T20:33:55.156309+0000","last_became_active":"2024-03-16T20:33:49.851188+0000","last_became_peered":"2024-03-16T20:33:49.851188+0000","last_unstale":"2024-03-16T20:33:55.156309+0000","last_undegraded":"2024-03-16T20:33:55.156309+0000","last_fullsized":"2024-03-16T20:33:55.156309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:34:00.153832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110027+0000","last_change":"2024-03-16T20:33:49.845150+0000","last_active":"2024-03-16T20:33:55.110027+0000","last_peered":"2024-03-16T20:33:55.110027+0000","last_clean":"2024-03-16T20:33:55.110027+0000","last_became_active":"2024-03-16T20:33:49.844531+0000","last_became_peered":"2024-03-16T20:33:49.844531+0000","last_unstale":"2024-03-16T20:33:55.110027+0000","last_undegraded":"2024-03-16T20:33:55.110027+0000","last_fullsized":"2024-03-16T20:33:55.110027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:54.607559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109405+0000","last_change":"2024-03-16T20:33:49.853608+0000","last_active":"2024-03-16T20:33:55.109405+0000","last_peered":"2024-03-16T20:33:55.109405+0000","last_clean":"2024-03-16T20:33:55.109405+0000","last_became_active":"2024-03-16T20:33:49.852800+0000","last_became_peered":"2024-03-16T20:33:49.852800+0000","last_unstale":"2024-03-16T20:33:55.109405+0000","last_undegraded":"2024-03-16T20:33:55.109405+0000","last_fullsized":"2024-03-16T20:33:55.109405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:59.988313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110862+0000","last_change":"2024-03-16T20:33:49.863542+0000","last_active":"2024-03-16T20:33:55.110862+0000","last_peered":"2024-03-16T20:33:55.110862+0000","last_clean":"2024-03-16T20:33:55.110862+0000","last_became_active":"2024-03-16T20:33:49.863143+0000","last_became_peered":"2024-03-16T20:33:49.863143+0000","last_unstale":"2024-03-16T20:33:55.110862+0000","last_undegraded":"2024-03-16T20:33:55.110862+0000","last_fullsized":"2024-03-16T20:33:55.110862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:18:29.997097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873507+0000","last_change":"2024-03-16T20:33:49.848285+0000","last_active":"2024-03-16T20:33:54.873507+0000","last_peered":"2024-03-16T20:33:54.873507+0000","last_clean":"2024-03-16T20:33:54.873507+0000","last_became_active":"2024-03-16T20:33:49.847942+0000","last_became_peered":"2024-03-16T20:33:49.847942+0000","last_unstale":"2024-03-16T20:33:54.873507+0000","last_undegraded":"2024-03-16T20:33:54.873507+0000","last_fullsized":"2024-03-16T20:33:54.873507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:08.287296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156564+0000","last_change":"2024-03-16T20:33:49.853004+0000","last_active":"2024-03-16T20:33:55.156564+0000","last_peered":"2024-03-16T20:33:55.156564+0000","last_clean":"2024-03-16T20:33:55.156564+0000","last_became_active":"2024-03-16T20:33:49.852609+0000","last_became_peered":"2024-03-16T20:33:49.852609+0000","last_unstale":"2024-03-16T20:33:55.156564+0000","last_undegraded":"2024-03-16T20:33:55.156564+0000","last_fullsized":"2024-03-16T20:33:55.156564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:42:58.648008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.109992+0000","last_change":"2024-03-16T20:33:49.857259+0000","last_active":"2024-03-16T20:33:55.109992+0000","last_peered":"2024-03-16T20:33:55.109992+0000","last_clean":"2024-03-16T20:33:55.109992+0000","last_became_active":"2024-03-16T20:33:49.856898+0000","last_became_peered":"2024-03-16T20:33:49.856898+0000","last_unstale":"2024-03-16T20:33:55.109992+0000","last_undegraded":"2024-03-16T20:33:55.109992+0000","last_fullsized":"2024-03-16T20:33:55.109992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:54.203804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110129+0000","last_change":"2024-03-16T20:33:49.860995+0000","last_active":"2024-03-16T20:33:55.110129+0000","last_peered":"2024-03-16T20:33:55.110129+0000","last_clean":"2024-03-16T20:33:55.110129+0000","last_became_active":"2024-03-16T20:33:49.860635+0000","last_became_peered":"2024-03-16T20:33:49.860635+0000","last_unstale":"2024-03-16T20:33:55.110129+0000","last_undegraded":"2024-03-16T20:33:55.110129+0000","last_fullsized":"2024-03-16T20:33:55.110129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:28:59.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.874468+0000","last_change":"2024-03-16T20:33:49.854841+0000","last_active":"2024-03-16T20:33:54.874468+0000","last_peered":"2024-03-16T20:33:54.874468+0000","last_clean":"2024-03-16T20:33:54.874468+0000","last_became_active":"2024-03-16T20:33:49.854468+0000","last_became_peered":"2024-03-16T20:33:49.854468+0000","last_unstale":"2024-03-16T20:33:54.874468+0000","last_undegraded":"2024-03-16T20:33:54.874468+0000","last_fullsized":"2024-03-16T20:33:54.874468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:52.224203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.110734+0000","last_change":"2024-03-16T20:33:49.854757+0000","last_active":"2024-03-16T20:33:55.110734+0000","last_peered":"2024-03-16T20:33:55.110734+0000","last_clean":"2024-03-16T20:33:55.110734+0000","last_became_active":"2024-03-16T20:33:49.852131+0000","last_became_peered":"2024-03-16T20:33:49.852131+0000","last_unstale":"2024-03-16T20:33:55.110734+0000","last_undegraded":"2024-03-16T20:33:55.110734+0000","last_fullsized":"2024-03-16T20:33:55.110734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:14.731341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.871423+0000","last_change":"2024-03-16T20:33:49.863332+0000","last_active":"2024-03-16T20:33:54.871423+0000","last_peered":"2024-03-16T20:33:54.871423+0000","last_clean":"2024-03-16T20:33:54.871423+0000","last_became_active":"2024-03-16T20:33:49.862953+0000","last_became_peered":"2024-03-16T20:33:49.862953+0000","last_unstale":"2024-03-16T20:33:54.871423+0000","last_undegraded":"2024-03-16T20:33:54.871423+0000","last_fullsized":"2024-03-16T20:33:54.871423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:17.126504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:54.873363+0000","last_change":"2024-03-16T20:33:49.849055+0000","last_active":"2024-03-16T20:33:54.873363+0000","last_peered":"2024-03-16T20:33:54.873363+0000","last_clean":"2024-03-16T20:33:54.873363+0000","last_became_active":"2024-03-16T20:33:49.848686+0000","last_became_peered":"2024-03-16T20:33:49.848686+0000","last_unstale":"2024-03-16T20:33:54.873363+0000","last_undegraded":"2024-03-16T20:33:54.873363+0000","last_fullsized":"2024-03-16T20:33:54.873363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:15.783993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-16T20:33:55.156397+0000","last_change":"2024-03-16T20:33:49.860146+0000","last_active":"2024-03-16T20:33:55.156397+0000","last_peered":"2024-03-16T20:33:55.156397+0000","last_clean":"2024-03-16T20:33:55.156397+0000","last_became_active":"2024-03-16T20:33:49.859768+0000","last_became_peered":"2024-03-16T20:33:49.859768+0000","last_unstale":"2024-03-16T20:33:55.156397+0000","last_undegraded":"2024-03-16T20:33:55.156397+0000","last_fullsized":"2024-03-16T20:33:55.156397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:20:10.747472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286596,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28975,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":37,"num_read_kb":34,"num_write":207,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":52151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":483005,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":1079342,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":484127,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":483743,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1075165,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":483117,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":495199,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":483095,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":488819,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483435,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":483671,"data_compressed":14666,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1074849,"data_compressed":51104,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":12194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:34:13.815 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-16T20:34:13.815 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T20:34:13.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T20:34:14.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:13 smithi023 ceph-mon[31879]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:13 smithi023 ceph-mon[31879]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:13 smithi023 ceph-mon[31879]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T20:34:14.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:13 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1830348029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:14.098 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:13 smithi031 ceph-mon[36281]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:13 smithi031 ceph-mon[36281]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:13 smithi031 ceph-mon[36281]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T20:34:14.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:13 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1830348029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:13 smithi189 ceph-mon[36655]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:13 smithi189 ceph-mon[36655]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:34:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:13 smithi189 ceph-mon[36655]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-16T20:34:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:13 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1830348029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:15.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:14 smithi023 ceph-mon[31879]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:14 smithi023 ceph-mon[31879]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:14 smithi023 ceph-mon[31879]: pgmap v302: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.9 KiB/s wr, 16 op/s 2024-03-16T20:34:15.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:14 smithi031 ceph-mon[36281]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:14 smithi031 ceph-mon[36281]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:14 smithi031 ceph-mon[36281]: pgmap v302: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.9 KiB/s wr, 16 op/s 2024-03-16T20:34:15.473 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:14 smithi189 ceph-mon[36655]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.473 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:14 smithi189 ceph-mon[36655]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:34:15.473 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:14 smithi189 ceph-mon[36655]: pgmap v302: 129 pgs: 129 active+clean; 605 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.9 KiB/s wr, 16 op/s 2024-03-16T20:34:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/170024113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd28102ac0 msgr2=0x7fbd28104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/170024113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd28102ac0 0x7fbd28104eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000b0a0 tx=0x7fbd2000b3b0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/170024113 shutdown_connections 2024-03-16T20:34:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/170024113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd28102ac0 0x7fbd28104eb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/170024113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 0x7fbd2810a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/170024113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd281053f0 0x7fbd281077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/170024113 >> 172.21.15.23:0/170024113 conn(0x7fbd280fc4d0 msgr2=0x7fbd280fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/170024113 shutdown_connections 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.862+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/170024113 wait complete. 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 Processor -- start 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 -- start start 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd28102ac0 0x7fbd28102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 0x7fbd28100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28101730 con 0x7fbd281053f0 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd281018a0 con 0x7fbd28102ac0 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd2d8f4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd281a2070 con 0x7fbd28107d20 2024-03-16T20:34:15.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd28102ac0 0x7fbd28102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60140/0 (socket says 172.21.15.23:60140) 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 -- 172.21.15.23:0/2374720130 learned_addr learned my addr 172.21.15.23:0/2374720130 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd277fe700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 0x7fbd28100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 -- 172.21.15.23:0/2374720130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 msgr2=0x7fbd28100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 0x7fbd28100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 -- 172.21.15.23:0/2374720130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd28102ac0 msgr2=0x7fbd28102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd28102ac0 0x7fbd28102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.863+0000 7fbd267fc700 1 -- 172.21.15.23:0/2374720130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20009d00 con 0x7fbd281053f0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd267fc700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e9a0 tx=0x7fbd1000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10009790 con 0x7fbd281053f0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd10010430 con 0x7fbd281053f0 2024-03-16T20:34:15.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd281a2350 con 0x7fbd281053f0 2024-03-16T20:34:15.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10018680 con 0x7fbd281053f0 2024-03-16T20:34:15.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.864+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd281a2850 con 0x7fbd281053f0 2024-03-16T20:34:15.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.865+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd280fe050 con 0x7fbd281053f0 2024-03-16T20:34:15.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.867+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbd10010910 con 0x7fbd281053f0 2024-03-16T20:34:15.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.867+0000 7fbd2c8f2700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 0x7fbd140a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:15.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.867+0000 7fbd26ffd700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 0x7fbd140a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:15.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.867+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbd10014070 con 0x7fbd281053f0 2024-03-16T20:34:15.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.868+0000 7fbd26ffd700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 0x7fbd140a40c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd20007ab0 tx=0x7fbd2000b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:15.871+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbd10096d40 con 0x7fbd281053f0 2024-03-16T20:34:16.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:15 smithi023 ceph-mon[31879]: pgmap v303: 129 pgs: 129 active+clean; 606 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T20:34:16.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.165+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fbd280504f0 con 0x7fbd281053f0 2024-03-16T20:34:16.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.167+0000 7fbd2c8f2700 1 -- 172.21.15.23:0/2374720130 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fbd100105a0 con 0x7fbd281053f0 2024-03-16T20:34:16.172 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-16T20:34:16.173 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBIAvZlpfnrCRAAG6jpATvzaXlqHxfEmA4VmA== 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 msgr2=0x7fbd140a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 0x7fbd140a40c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd20007ab0 tx=0x7fbd2000b700 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 msgr2=0x7fbd28100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000e9a0 tx=0x7fbd1000ecb0 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 shutdown_connections 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbd140a1c00 0x7fbd140a40c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd28107d20 0x7fbd28100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd28102ac0 0x7fbd28102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 --2- 172.21.15.23:0/2374720130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd281053f0 0x7fbd28100a70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.169+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 >> 172.21.15.23:0/2374720130 conn(0x7fbd280fc4d0 msgr2=0x7fbd28103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.170+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 shutdown_connections 2024-03-16T20:34:16.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:16.170+0000 7fbd2d8f4700 1 -- 172.21.15.23:0/2374720130 wait complete. 2024-03-16T20:34:16.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:15 smithi031 ceph-mon[36281]: pgmap v303: 129 pgs: 129 active+clean; 606 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T20:34:16.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:15 smithi189 ceph-mon[36655]: pgmap v303: 129 pgs: 129 active+clean; 606 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-03-16T20:34:16.893 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:34:16.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T20:34:16.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T20:34:16.936 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T20:34:16.936 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T20:34:16.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T20:34:17.221 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:17.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:16 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T20:34:17.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:16 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T20:34:17.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:16 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T20:34:17.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:16 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T20:34:17.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:16 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T20:34:17.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:16 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2374720130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T20:34:18.295 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:17 smithi023 ceph-mon[31879]: pgmap v304: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-16T20:34:18.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:17 smithi031 ceph-mon[36281]: pgmap v304: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-16T20:34:18.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:17 smithi189 ceph-mon[36655]: pgmap v304: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.807+0000 7f7a8cb32700 1 -- 172.21.15.23:0/42408636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806a940 msgr2=0x7f7a8806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.807+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/42408636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806a940 0x7f7a8806ad20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009990 tx=0x7f7a7c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 -- 172.21.15.23:0/42408636 shutdown_connections 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/42408636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806a940 0x7f7a8806ad20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/42408636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 0x7f7a8810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/42408636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806b2f0 0x7f7a88106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 -- 172.21.15.23:0/42408636 >> 172.21.15.23:0/42408636 conn(0x7f7a88076f00 msgr2=0x7f7a88077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 -- 172.21.15.23:0/42408636 shutdown_connections 2024-03-16T20:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 -- 172.21.15.23:0/42408636 wait complete. 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 Processor -- start 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.808+0000 7f7a8cb32700 1 -- start start 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 0x7f7a88101520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 0x7f7a88101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881a6a00 con 0x7f7a8806b2f0 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881a6b70 con 0x7f7a8806a940 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8cb32700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881a6d30 con 0x7f7a88107360 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a85d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 0x7f7a88101520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:45318/0 (socket says 172.21.15.23:45318) 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 -- 172.21.15.23:0/2534679429 learned_addr learned my addr 172.21.15.23:0/2534679429 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:18.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a85d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 0x7f7a88101520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60146/0 (socket says 172.21.15.23:60146) 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a86d9d700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 0x7f7a88101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 -- 172.21.15.23:0/2534679429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 msgr2=0x7f7a88101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 0x7f7a88101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 -- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 msgr2=0x7f7a88101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 0x7f7a88101520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 -- 172.21.15.23:0/2534679429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0095f0 con 0x7f7a8806a940 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a8659c700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c005910 tx=0x7f7a7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.809+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c01d070 con 0x7f7a8806a940 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.810+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a7c00b610 con 0x7f7a8806a940 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.810+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a881a6fb0 con 0x7f7a8806a940 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.810+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0173f0 con 0x7f7a8806a940 2024-03-16T20:34:18.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.810+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881a7360 con 0x7f7a8806a940 2024-03-16T20:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.811+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8810a510 con 0x7f7a8806a940 2024-03-16T20:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.812+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7a7c0225e0 con 0x7f7a8806a940 2024-03-16T20:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.812+0000 7f7a777fe700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 0x7f7a700a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.812+0000 7f7a85d9b700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 0x7f7a700a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.812+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7a7c0ce9b0 con 0x7f7a8806a940 2024-03-16T20:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.813+0000 7f7a85d9b700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 0x7f7a700a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a88100b80 tx=0x7f7a78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:18.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:18.817+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7a7c0978b0 con 0x7f7a8806a940 2024-03-16T20:34:19.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.077+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7a881120b0 con 0x7f7a8806a940 2024-03-16T20:34:19.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.086+0000 7f7a777fe700 1 -- 172.21.15.23:0/2534679429 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260660 (secure 0 0 0) 0x7f7a7c09af70 con 0x7f7a8806a940 2024-03-16T20:34:19.095 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 msgr2=0x7f7a700a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 0x7f7a700a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a88100b80 tx=0x7f7a78009260 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 msgr2=0x7f7a88100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c005910 tx=0x7f7a7c004730 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 shutdown_connections 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7a700a1b50 0x7f7a700a4010 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a88107360 0x7f7a88101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a8806a940 0x7f7a88100fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 --2- 172.21.15.23:0/2534679429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a8806b2f0 0x7f7a88101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 >> 172.21.15.23:0/2534679429 conn(0x7f7a88076f00 msgr2=0x7f7a880ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 shutdown_connections 2024-03-16T20:34:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:19.089+0000 7f7a8cb32700 1 -- 172.21.15.23:0/2534679429 wait complete. 2024-03-16T20:34:19.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:19 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2534679429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T20:34:19.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:19 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2534679429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T20:34:19.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:19 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2534679429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T20:34:19.843 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":3040096369},{"type":"v1","addr":"172.21.15.23:6801","nonce":3040096369}]},"active_addr":"172.21.15.23:6801/3040096369","active_change":"2024-03-16T20:26:49.857080+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3105884864}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3890971788}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":885364000}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1087781272}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":22442756}]}]} 2024-03-16T20:34:19.847 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T20:34:19.848 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T20:34:19.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T20:34:20.094 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:20 smithi031 ceph-mon[36281]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-03-16T20:34:20.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:20 smithi023 ceph-mon[31879]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-03-16T20:34:20.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:20 smithi189 ceph-mon[36655]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 -- 172.21.15.23:0/475541740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e0108030 msgr2=0x7fc6e0108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/475541740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e0108030 0x7fc6e0108410 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00099c0 tx=0x7fc6d0009cd0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 -- 172.21.15.23:0/475541740 shutdown_connections 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/475541740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e0108030 0x7fc6e0108410 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/475541740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e006a630 0x7fc6e010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/475541740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 0x7fc6e006a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 -- 172.21.15.23:0/475541740 >> 172.21.15.23:0/475541740 conn(0x7fc6e0077e10 msgr2=0x7fc6e0078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 -- 172.21.15.23:0/475541740 shutdown_connections 2024-03-16T20:34:21.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.850+0000 7fc6e67a1700 1 -- 172.21.15.23:0/475541740 wait complete. 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 Processor -- start 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 -- start start 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 0x7fc6e0100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e0108030 0x7fc6e0101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e01a68a0 con 0x7fc6e006a630 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e01a6a10 con 0x7fc6e0108030 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e67a1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e01a6bd0 con 0x7fc6e0069c90 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60580/0 (socket says 172.21.15.23:60580) 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6df7fe700 1 -- 172.21.15.23:0/2391939226 learned_addr learned my addr 172.21.15.23:0/2391939226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6dffff700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 0x7fc6e0100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:21.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.851+0000 7fc6e4d3e700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e0108030 0x7fc6e0101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 -- 172.21.15.23:0/2391939226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 msgr2=0x7fc6e0100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 0x7fc6e0100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 -- 172.21.15.23:0/2391939226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e0108030 msgr2=0x7fc6e0101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e0108030 0x7fc6e0101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 -- 172.21.15.23:0/2391939226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00095f0 con 0x7fc6e006a630 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6df7fe700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e9a0 tx=0x7fc6d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4009790 con 0x7fc6e006a630 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d4010430 con 0x7fc6e006a630 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e01a6eb0 con 0x7fc6e006a630 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4018680 con 0x7fc6e006a630 2024-03-16T20:34:21.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.852+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e01a7320 con 0x7fc6e006a630 2024-03-16T20:34:21.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.853+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e010a550 con 0x7fc6e006a630 2024-03-16T20:34:21.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.854+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc6d4010910 con 0x7fc6e006a630 2024-03-16T20:34:21.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.855+0000 7fc6dd7fa700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 0x7fc6c80a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:21.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.855+0000 7fc6dffff700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 0x7fc6c80a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:21.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.855+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc6d4014070 con 0x7fc6e006a630 2024-03-16T20:34:21.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.855+0000 7fc6dffff700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 0x7fc6c80a4010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0000c00 tx=0x7fc6d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:21.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:21.859+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc6d4053280 con 0x7fc6e006a630 2024-03-16T20:34:22.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.066+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e0111ff0 con 0x7fc6e006a630 2024-03-16T20:34:22.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.067+0000 7fc6dd7fa700 1 -- 172.21.15.23:0/2391939226 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fc6d4096850 con 0x7fc6e006a630 2024-03-16T20:34:22.073 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:22.073 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:22.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 msgr2=0x7fc6c80a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:22.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 0x7fc6c80a4010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0000c00 tx=0x7fc6d0005fb0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 msgr2=0x7fc6e01014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e9a0 tx=0x7fc6d400ecb0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 shutdown_connections 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c80a1b50 0x7fc6c80a4010 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6e0108030 0x7fc6e0101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6e0069c90 0x7fc6e0100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 --2- 172.21.15.23:0/2391939226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e006a630 0x7fc6e01014f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 >> 172.21.15.23:0/2391939226 conn(0x7fc6e0077e10 msgr2=0x7fc6e00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 shutdown_connections 2024-03-16T20:34:22.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:22.069+0000 7fc6e67a1700 1 -- 172.21.15.23:0/2391939226 wait complete. 2024-03-16T20:34:22.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:22 smithi023 ceph-mon[31879]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 26 op/s 2024-03-16T20:34:22.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:22 smithi031 ceph-mon[36281]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 26 op/s 2024-03-16T20:34:22.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:22 smithi189 ceph-mon[36655]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 26 op/s 2024-03-16T20:34:22.854 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T20:34:22.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T20:34:23.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:23 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2391939226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:23.099 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:23.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:23 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2391939226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:23.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:23 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2391939226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:24.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:24 smithi023 ceph-mon[31879]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-03-16T20:34:24.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:24 smithi031 ceph-mon[36281]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-03-16T20:34:24.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:24 smithi189 ceph-mon[36655]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.876+0000 7f8009bc2700 1 -- 172.21.15.23:0/2056692162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8004102fc0 msgr2=0x7f800406a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.876+0000 7f8009bc2700 1 --2- 172.21.15.23:0/2056692162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8004102fc0 0x7f800406a720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009990 tx=0x7f7ff4009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 -- 172.21.15.23:0/2056692162 shutdown_connections 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 --2- 172.21.15.23:0/2056692162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8004102fc0 0x7f800406a720 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 --2- 172.21.15.23:0/2056692162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f800410e960 0x7f8004110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 --2- 172.21.15.23:0/2056692162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f800406b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 -- 172.21.15.23:0/2056692162 >> 172.21.15.23:0/2056692162 conn(0x7f8004077d40 msgr2=0x7f8004078150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:24.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 -- 172.21.15.23:0/2056692162 shutdown_connections 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 -- 172.21.15.23:0/2056692162 wait complete. 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 Processor -- start 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.877+0000 7f8009bc2700 1 -- start start 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f80041a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 0x7f80041a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800419ac60 con 0x7f800410e960 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800419ada0 con 0x7f800406acf0 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800419af60 con 0x7f8004102fc0 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:24.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60594/0 (socket says 172.21.15.23:60594) 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f80037fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f80041a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 -- 172.21.15.23:0/342304975 learned_addr learned my addr 172.21.15.23:0/342304975 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8002ffd700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 0x7f80041a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 -- 172.21.15.23:0/342304975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 msgr2=0x7f80041a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 0x7f80041a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 -- 172.21.15.23:0/342304975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 msgr2=0x7f80041a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f80041a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 -- 172.21.15.23:0/342304975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40095f0 con 0x7f800410e960 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8002ffd700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 0x7f80041a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f80037fe700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f80041a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8003fff700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00efc0 tx=0x7f7ffc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc018070 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800419b1e0 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ffc00f040 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00cd40 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.878+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80041a7490 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.879+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8004068ca0 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.881+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7ffc003a00 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.881+0000 7f8000ff9700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 0x7f7fec0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.881+0000 7f80037fe700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 0x7f7fec0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.882+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7ffc0cdb50 con 0x7f800410e960 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.882+0000 7f80037fe700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 0x7f7fec0a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400b2e0 tx=0x7f7ff4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:24.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:24.885+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7ffc0969f0 con 0x7f800410e960 2024-03-16T20:34:25.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.090+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80041142e0 con 0x7f800410e960 2024-03-16T20:34:25.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.092+0000 7f8000ff9700 1 -- 172.21.15.23:0/342304975 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f7ffc09a0b0 con 0x7f800410e960 2024-03-16T20:34:25.097 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:34:25.098 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:33:54.239218+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:34:25.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 msgr2=0x7f7fec0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:25.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 0x7f7fec0a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400b2e0 tx=0x7f7ff4005960 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 msgr2=0x7f800419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00efc0 tx=0x7f7ffc00c3a0 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 shutdown_connections 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f7fec0a1b50 0x7f7fec0a4010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f800410e960 0x7f800419a600 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f800406acf0 0x7f80041a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 --2- 172.21.15.23:0/342304975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8004102fc0 0x7f80041a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 >> 172.21.15.23:0/342304975 conn(0x7f8004077d40 msgr2=0x7f80041001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 shutdown_connections 2024-03-16T20:34:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:25.094+0000 7f8009bc2700 1 -- 172.21.15.23:0/342304975 wait complete. 2024-03-16T20:34:25.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:25 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/342304975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:25.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:25 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/342304975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:25.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:25 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/342304975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:34:25.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T20:34:25.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T20:34:25.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T20:34:25.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T20:34:25.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T20:34:25.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T20:34:25.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T20:34:25.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T20:34:25.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-16T20:34:25.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-16T20:34:25.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-16T20:34:25.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-16T20:34:26.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:26 smithi031 ceph-mon[36281]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-03-16T20:34:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:26 smithi189 ceph-mon[36655]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-03-16T20:34:26.545 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:26.555 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:26.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:26 smithi023 ceph-mon[31879]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-03-16T20:34:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.576 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.577 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.578 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:27.579 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:28.010 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:28.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:28 smithi023 ceph-mon[31879]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-16T20:34:28.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:28 smithi031 ceph-mon[36281]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-16T20:34:28.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:28 smithi189 ceph-mon[36655]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-16T20:34:30.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:30 smithi023 ceph-mon[31879]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T20:34:30.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:30 smithi031 ceph-mon[36281]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T20:34:30.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:30 smithi189 ceph-mon[36655]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-16T20:34:32.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:32 smithi023 ceph-mon[31879]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T20:34:32.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:32 smithi031 ceph-mon[36281]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T20:34:32.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:32 smithi189 ceph-mon[36655]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 -- 172.21.15.23:0/2112088562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 msgr2=0x7f1c4c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 --2- 172.21.15.23:0/2112088562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c10ee70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400b0d0 tx=0x7f1c4400b3e0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 -- 172.21.15.23:0/2112088562 shutdown_connections 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 --2- 172.21.15.23:0/2112088562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c10ee70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 --2- 172.21.15.23:0/2112088562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c108030 0x7f1c4c108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 --2- 172.21.15.23:0/2112088562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c069c90 0x7f1c4c06a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 -- 172.21.15.23:0/2112088562 >> 172.21.15.23:0/2112088562 conn(0x7f1c4c077e10 msgr2=0x7f1c4c078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.333+0000 7f1c52317700 1 -- 172.21.15.23:0/2112088562 shutdown_connections 2024-03-16T20:34:33.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 -- 172.21.15.23:0/2112088562 wait complete. 2024-03-16T20:34:33.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 Processor -- start 2024-03-16T20:34:33.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 -- start start 2024-03-16T20:34:33.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c069c90 0x7f1c4c100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:33.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c1014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c108030 0x7f1c4c101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a6a00 con 0x7f1c4c06a630 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a6b70 con 0x7f1c4c108030 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.334+0000 7f1c52317700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1a6d30 con 0x7f1c4c069c90 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c1014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c108030 0x7f1c4c101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c1014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35046/0 (socket says 172.21.15.23:35046) 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4b7fe700 1 -- 172.21.15.23:0/2421585133 learned_addr learned my addr 172.21.15.23:0/2421585133 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4bfff700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c069c90 0x7f1c4c100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 -- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c069c90 msgr2=0x7f1c4c100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c069c90 0x7f1c4c100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 -- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 msgr2=0x7f1c4c1014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c1014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44009d00 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4bfff700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c4c069c90 0x7f1c4c100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c4b7fe700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c4c06a630 0x7f1c4c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c508b4700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c108030 0x7f1c4c101a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400d010 tx=0x7f1c4400bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4400e040 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c1a6fb0 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.335+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c440079d0 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.336+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c1a7420 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.336+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4401d480 con 0x7f1c4c108030 2024-03-16T20:34:33.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.336+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c38001030 con 0x7f1c4c108030 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.338+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1c44005090 con 0x7f1c4c108030 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c497fa700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1c340a1b50 0x7f1c340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c4bfff700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1c340a1b50 0x7f1c340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1c440ce730 con 0x7f1c4c108030 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c497fa700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] conn(0x7f1c340ac810 0x7f1c340aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c340af320 con 0x7f1c340ac810 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c440ceaa0 con 0x7f1c4c108030 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c4bfff700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1c340a1b50 0x7f1c340a4010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009fd0 tx=0x7f1c3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.339+0000 7f1c4b7fe700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] conn(0x7f1c340ac810 0x7f1c340aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:33.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.340+0000 7f1c4b7fe700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] conn(0x7f1c340ac810 0x7f1c340aec50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:33.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.340+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== osd.5 v2:172.21.15.189:6808/2617907376 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1c340af320 con 0x7f1c340ac810 2024-03-16T20:34:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.361+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 --> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c38002dc0 con 0x7f1c340ac810 2024-03-16T20:34:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c497fa700 1 -- 172.21.15.23:0/2421585133 <== osd.5 v2:172.21.15.189:6808/2617907376 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1c38002dc0 con 0x7f1c340ac810 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] conn(0x7f1c340ac810 msgr2=0x7f1c340aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] conn(0x7f1c340ac810 0x7f1c340aec50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1c340a1b50 msgr2=0x7f1c340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1c340a1b50 0x7f1c340a4010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009fd0 tx=0x7f1c3c009260 comp rx=0 tx=0).stop 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c108030 msgr2=0x7f1c4c101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 --2- 172.21.15.23:0/2421585133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c4c108030 0x7f1c4c101a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400d010 tx=0x7f1c4400bcd0 comp rx=0 tx=0).stop 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c508b4700 1 -- 172.21.15.23:0/2421585133 reap_dead start 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 shutdown_connections 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 >> 172.21.15.23:0/2421585133 conn(0x7f1c4c077e10 msgr2=0x7f1c4c0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 shutdown_connections 2024-03-16T20:34:33.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:33.362+0000 7f1c52317700 1 -- 172.21.15.23:0/2421585133 wait complete. 2024-03-16T20:34:34.325 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:34 smithi023 ceph-mon[31879]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:34:34.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:34 smithi031 ceph-mon[36281]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:34:34.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:34 smithi189 ceph-mon[36655]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:34:36.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:36 smithi023 ceph-mon[31879]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:36.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:36 smithi031 ceph-mon[36281]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:36.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:36 smithi189 ceph-mon[36655]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:37.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 -- 172.21.15.23:0/3147166273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 msgr2=0x7f97d406a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 --2- 172.21.15.23:0/3147166273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d406a0f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f97c80099c0 tx=0x7f97c8009cd0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 -- 172.21.15.23:0/3147166273 shutdown_connections 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 --2- 172.21.15.23:0/3147166273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d406a630 0x7f97d410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 --2- 172.21.15.23:0/3147166273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d4108030 0x7f97d4108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 --2- 172.21.15.23:0/3147166273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d406a0f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 -- 172.21.15.23:0/3147166273 >> 172.21.15.23:0/3147166273 conn(0x7f97d4077e10 msgr2=0x7f97d4078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 -- 172.21.15.23:0/3147166273 shutdown_connections 2024-03-16T20:34:37.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.348+0000 7f97db371700 1 -- 172.21.15.23:0/3147166273 wait complete. 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 Processor -- start 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 -- start start 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 0x7f97d4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d4108030 0x7f97d419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419d420 con 0x7f97d4069c90 2024-03-16T20:34:37.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419d590 con 0x7f97d4108030 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97db371700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419d750 con 0x7f97d406a630 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d910d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d990e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d4108030 0x7f97d419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d890c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 0x7f97d4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d910d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d4198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35064/0 (socket says 172.21.15.23:35064) 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d910d700 1 -- 172.21.15.23:0/3102450171 learned_addr learned my addr 172.21.15.23:0/3102450171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.349+0000 7f97d890c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 0x7f97d4198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:49550/0 (socket says 172.21.15.23:49550) 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 -- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 msgr2=0x7f97d4198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 0x7f97d4198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 -- 172.21.15.23:0/3102450171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d4108030 msgr2=0x7f97d419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d4108030 0x7f97d419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c80095f0 con 0x7f97d4069c90 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d890c700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d406a630 0x7f97d4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:37.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d990e700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97d4108030 0x7f97d419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d910d700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d4198540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ef10 tx=0x7f97c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c400c9c0 con 0x7f97d4069c90 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d419d9d0 con 0x7f97d4069c90 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97c400f040 con 0x7f97d4069c90 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4014480 con 0x7f97d4069c90 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.350+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d41a5290 con 0x7f97d4069c90 2024-03-16T20:34:37.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.351+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97d41122e0 con 0x7f97d4069c90 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.352+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f97c4007720 con 0x7f97d4069c90 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d27fc700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97bc0a1b50 0x7f97bc0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d890c700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97bc0a1b50 0x7f97bc0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f97c40ce480 con 0x7f97d4069c90 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d27fc700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] conn(0x7f97bc0ac810 0x7f97bc0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97bc0af320 con 0x7f97bc0ac810 2024-03-16T20:34:37.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97c40ce890 con 0x7f97d4069c90 2024-03-16T20:34:37.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d890c700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97bc0a1b50 0x7f97bc0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.353+0000 7f97d990e700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] conn(0x7f97bc0ac810 0x7f97bc0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.354+0000 7f97d990e700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] conn(0x7f97bc0ac810 0x7f97bc0aec50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.354+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== osd.2 v2:172.21.15.189:6800/3710574436 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f97bc0af320 con 0x7f97bc0ac810 2024-03-16T20:34:37.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.371+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 --> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f97d40626f0 con 0x7f97bc0ac810 2024-03-16T20:34:37.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.371+0000 7f97d27fc700 1 -- 172.21.15.23:0/3102450171 <== osd.2 v2:172.21.15.189:6800/3710574436 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f97d40626f0 con 0x7f97bc0ac810 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] conn(0x7f97bc0ac810 msgr2=0x7f97bc0aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] conn(0x7f97bc0ac810 0x7f97bc0aec50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97bc0a1b50 msgr2=0x7f97bc0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97bc0a1b50 0x7f97bc0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8005fb0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 msgr2=0x7f97d4198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 --2- 172.21.15.23:0/3102450171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d4069c90 0x7f97d4198540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ef10 tx=0x7f97c400c3a0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97d990e700 1 -- 172.21.15.23:0/3102450171 reap_dead start 2024-03-16T20:34:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 shutdown_connections 2024-03-16T20:34:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 >> 172.21.15.23:0/3102450171 conn(0x7f97d4077e10 msgr2=0x7f97d40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 shutdown_connections 2024-03-16T20:34:37.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:37.372+0000 7f97db371700 1 -- 172.21.15.23:0/3102450171 wait complete. 2024-03-16T20:34:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 -- 172.21.15.23:0/225576240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 msgr2=0x7fc878105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 --2- 172.21.15.23:0/225576240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc878105b80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc864009990 tx=0x7fc864009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 -- 172.21.15.23:0/225576240 shutdown_connections 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 --2- 172.21.15.23:0/225576240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc878106260 0x7fc87810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 --2- 172.21.15.23:0/225576240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc878101940 0x7fc878101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 --2- 172.21.15.23:0/225576240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc878105b80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 -- 172.21.15.23:0/225576240 >> 172.21.15.23:0/225576240 conn(0x7fc8780774e0 msgr2=0x7fc8780fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 -- 172.21.15.23:0/225576240 shutdown_connections 2024-03-16T20:34:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.511+0000 7fc87eb46700 1 -- 172.21.15.23:0/225576240 wait complete. 2024-03-16T20:34:37.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 Processor -- start 2024-03-16T20:34:37.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 -- start start 2024-03-16T20:34:37.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc878101940 0x7fc878113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc8781118d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc878106260 0x7fc878111e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878112590 con 0x7fc8781022f0 2024-03-16T20:34:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878112700 con 0x7fc878106260 2024-03-16T20:34:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87eb46700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8781a6c70 con 0x7fc878101940 2024-03-16T20:34:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc8781118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc8781118d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35054/0 (socket says 172.21.15.23:35054) 2024-03-16T20:34:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 -- 172.21.15.23:0/4281080602 learned_addr learned my addr 172.21.15.23:0/4281080602 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87d0e3700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc878106260 0x7fc878111e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc87c8e2700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc878101940 0x7fc878113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 -- 172.21.15.23:0/4281080602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc878101940 msgr2=0x7fc878113350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc878101940 0x7fc878113350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 -- 172.21.15.23:0/4281080602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc878106260 msgr2=0x7fc878111e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc878106260 0x7fc878111e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.512+0000 7fc877fff700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640095f0 con 0x7fc8781022f0 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc877fff700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc8781118d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc864000c00 tx=0x7fc8640057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86401d070 con 0x7fc8781022f0 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8781a6ea0 con 0x7fc8781022f0 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc86400b610 con 0x7fc8781022f0 2024-03-16T20:34:37.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8640173f0 con 0x7fc8781022f0 2024-03-16T20:34:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.513+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8781a72c0 con 0x7fc8781022f0 2024-03-16T20:34:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.514+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc878111260 con 0x7fc8781022f0 2024-03-16T20:34:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.516+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc864021610 con 0x7fc8781022f0 2024-03-16T20:34:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.516+0000 7fc875ffb700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8680a1b50 0x7fc8680a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.516+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc8640cf270 con 0x7fc8781022f0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.516+0000 7fc87c8e2700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8680a1b50 0x7fc8680a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc875ffb700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] conn(0x7fc8680ac810 0x7fc8680aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8680af320 con 0x7fc8680ac810 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8640cf680 con 0x7fc8781022f0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc87d0e3700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] conn(0x7fc8680ac810 0x7fc8680aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc87c8e2700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8680a1b50 0x7fc8680a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009c60 tx=0x7fc86c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.517+0000 7fc87d0e3700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] conn(0x7fc8680ac810 0x7fc8680aec50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.518+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== osd.0 v2:172.21.15.23:6802/3821836785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc8680af320 con 0x7fc8680ac810 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.535+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 --> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc878105550 con 0x7fc8680ac810 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.535+0000 7fc875ffb700 1 -- 172.21.15.23:0/4281080602 <== osd.0 v2:172.21.15.23:6802/3821836785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc878105550 con 0x7fc8680ac810 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] conn(0x7fc8680ac810 msgr2=0x7fc8680aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] conn(0x7fc8680ac810 0x7fc8680aec50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8680a1b50 msgr2=0x7fc8680a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8680a1b50 0x7fc8680a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009c60 tx=0x7fc86c009260 comp rx=0 tx=0).stop 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 msgr2=0x7fc8781118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 --2- 172.21.15.23:0/4281080602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8781022f0 0x7fc8781118d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc864000c00 tx=0x7fc8640057c0 comp rx=0 tx=0).stop 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87d0e3700 1 -- 172.21.15.23:0/4281080602 reap_dead start 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 shutdown_connections 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 >> 172.21.15.23:0/4281080602 conn(0x7fc8780774e0 msgr2=0x7fc8781075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 shutdown_connections 2024-03-16T20:34:37.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:34.536+0000 7fc87eb46700 1 -- 172.21.15.23:0/4281080602 wait complete. 2024-03-16T20:34:38.083 INFO:teuthology.orchestra.run.smithi023.stdout:34359738446 2024-03-16T20:34:38.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T20:34:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:38 smithi023 ceph-mon[31879]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:38.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:38 smithi031 ceph-mon[36281]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:38.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:38 smithi189 ceph-mon[36655]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-16T20:34:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 -- 172.21.15.23:0/3192573115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 msgr2=0x7f8d8010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 --2- 172.21.15.23:0/3192573115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d8010ee30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800b0d0 tx=0x7f8d7800b3e0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 -- 172.21.15.23:0/3192573115 shutdown_connections 2024-03-16T20:34:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 --2- 172.21.15.23:0/3192573115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d8010ee30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 --2- 172.21.15.23:0/3192573115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80108010 0x7f8d801083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 --2- 172.21.15.23:0/3192573115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d801024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 -- 172.21.15.23:0/3192573115 >> 172.21.15.23:0/3192573115 conn(0x7f8d80079110 msgr2=0x7f8d80079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.552+0000 7f8d85538700 1 -- 172.21.15.23:0/3192573115 shutdown_connections 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 -- 172.21.15.23:0/3192573115 wait complete. 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 Processor -- start 2024-03-16T20:34:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 -- start start 2024-03-16T20:34:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d800742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d80074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.553+0000 7f8d85538700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80108010 0x7f8d80074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d85538700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801a6960 con 0x7f8d801029f0 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d85538700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801a6ad0 con 0x7f8d80102030 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d85538700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d801a6c90 con 0x7f8d80108010 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d80074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d800742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d800742e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:37446/0 (socket says 172.21.15.23:37446) 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 -- 172.21.15.23:0/2397905838 learned_addr learned my addr 172.21.15.23:0/2397905838 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7f7fe700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80108010 0x7f8d80074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 -- 172.21.15.23:0/2397905838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80108010 msgr2=0x7f8d80074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d80108010 0x7f8d80074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 -- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 msgr2=0x7f8d80074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d80074820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009d00 con 0x7f8d80102030 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7e7fc700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d801029f0 0x7f8d80074820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d7effd700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d800742e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000d950 tx=0x7f8d7000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000f890 con 0x7f8d80102030 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d70010460 con 0x7f8d80102030 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d801a6f70 con 0x7f8d80102030 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000e610 con 0x7f8d80102030 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.554+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d801a73e0 con 0x7f8d80102030 2024-03-16T20:34:38.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.555+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d801036c0 con 0x7f8d80102030 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.557+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f8d70010960 con 0x7f8d80102030 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.557+0000 7f8d67fff700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d680a1b50 0x7f8d680a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.557+0000 7f8d7e7fc700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d680a1b50 0x7f8d680a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.557+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8d700cda20 con 0x7f8d80102030 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d67fff700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] conn(0x7f8d680ac810 0x7f8d680aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d680af320 con 0x7f8d680ac810 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d700cdd90 con 0x7f8d80102030 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d7f7fe700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] conn(0x7f8d680ac810 0x7f8d680aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:38.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d7e7fc700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d680a1b50 0x7f8d680a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d800796a0 tx=0x7f8d74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:38.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.558+0000 7f8d7f7fe700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] conn(0x7f8d680ac810 0x7f8d680aec50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:38.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.559+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== osd.3 v2:172.21.15.23:6810/1075234291 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d680af320 con 0x7f8d680ac810 2024-03-16T20:34:38.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.575+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 --> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d80112320 con 0x7f8d680ac810 2024-03-16T20:34:38.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d67fff700 1 -- 172.21.15.23:0/2397905838 <== osd.3 v2:172.21.15.23:6810/1075234291 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8d80112320 con 0x7f8d680ac810 2024-03-16T20:34:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] conn(0x7f8d680ac810 msgr2=0x7f8d680aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] conn(0x7f8d680ac810 0x7f8d680aec50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d680a1b50 msgr2=0x7f8d680a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d680a1b50 0x7f8d680a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d800796a0 tx=0x7f8d74009260 comp rx=0 tx=0).stop 2024-03-16T20:34:38.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 msgr2=0x7f8d800742e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 --2- 172.21.15.23:0/2397905838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d80102030 0x7f8d800742e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000d950 tx=0x7f8d7000dc60 comp rx=0 tx=0).stop 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d7f7fe700 1 -- 172.21.15.23:0/2397905838 reap_dead start 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 shutdown_connections 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 >> 172.21.15.23:0/2397905838 conn(0x7f8d80079110 msgr2=0x7f8d80100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 shutdown_connections 2024-03-16T20:34:38.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:38.576+0000 7f8d85538700 1 -- 172.21.15.23:0/2397905838 wait complete. 2024-03-16T20:34:39.205 INFO:teuthology.orchestra.run.smithi023.stdout:107374182464 2024-03-16T20:34:39.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T20:34:40.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:40 smithi023 ceph-mon[31879]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-16T20:34:40.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 -- 172.21.15.23:0/2234056111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b006a630 msgr2=0x7f84b010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 --2- 172.21.15.23:0/2234056111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b006a630 0x7f84b010ee70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0099c0 tx=0x7f84ac009cd0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 -- 172.21.15.23:0/2234056111 shutdown_connections 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 --2- 172.21.15.23:0/2234056111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b006a630 0x7f84b010ee70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 --2- 172.21.15.23:0/2234056111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84b0108030 0x7f84b0108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 --2- 172.21.15.23:0/2234056111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b006a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 -- 172.21.15.23:0/2234056111 >> 172.21.15.23:0/2234056111 conn(0x7f84b0077e10 msgr2=0x7f84b0078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 -- 172.21.15.23:0/2234056111 shutdown_connections 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.298+0000 7f84b75ec700 1 -- 172.21.15.23:0/2234056111 wait complete. 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 Processor -- start 2024-03-16T20:34:40.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 -- start start 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84b006a630 0x7f84b0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b0108030 0x7f84b019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d3c0 con 0x7f84b0108030 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d530 con 0x7f84b0069c90 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b75ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d6f0 con 0x7f84b006a630 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5b89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b0108030 0x7f84b019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:37458/0 (socket says 172.21.15.23:37458) 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 -- 172.21.15.23:0/43606774 learned_addr learned my addr 172.21.15.23:0/43606774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b4b87700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84b006a630 0x7f84b0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 -- 172.21.15.23:0/43606774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84b006a630 msgr2=0x7f84b0198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84b006a630 0x7f84b0198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 -- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b0108030 msgr2=0x7f84b019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84b0108030 0x7f84b019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.299+0000 7f84b5388700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac0095f0 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84b5388700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b0198530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84a800ef10 tx=0x7f84a800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a800c9c0 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84a800f040 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b019d970 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8014480 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.300+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b01010b0 con 0x7f84b0069c90 2024-03-16T20:34:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.301+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84b0112290 con 0x7f84b0069c90 2024-03-16T20:34:40.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.303+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f84a8007720 con 0x7f84b0069c90 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.303+0000 7f84a67fc700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f849c0a1b50 0x7f849c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84b4b87700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f849c0a1b50 0x7f849c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f84a80ce480 con 0x7f84b0069c90 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84a67fc700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] conn(0x7f849c0ac810 0x7f849c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f849c0af320 con 0x7f849c0ac810 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84a80ce890 con 0x7f84b0069c90 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84b5b89700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] conn(0x7f849c0ac810 0x7f849c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84b4b87700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f849c0a1b50 0x7f849c0a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84b00783f0 tx=0x7f84a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.304+0000 7f84b5b89700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] conn(0x7f849c0ac810 0x7f849c0aec50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.305+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== osd.1 v2:172.21.15.31:6800/639380907 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f849c0af320 con 0x7f849c0ac810 2024-03-16T20:34:40.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.322+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 --> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f84b00626a0 con 0x7f849c0ac810 2024-03-16T20:34:40.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.322+0000 7f84a67fc700 1 -- 172.21.15.23:0/43606774 <== osd.1 v2:172.21.15.31:6800/639380907 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f84b00626a0 con 0x7f849c0ac810 2024-03-16T20:34:40.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] conn(0x7f849c0ac810 msgr2=0x7f849c0aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] conn(0x7f849c0ac810 0x7f849c0aec50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f849c0a1b50 msgr2=0x7f849c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f849c0a1b50 0x7f849c0a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84b00783f0 tx=0x7f84a0009260 comp rx=0 tx=0).stop 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 msgr2=0x7f84b0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 --2- 172.21.15.23:0/43606774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b0069c90 0x7f84b0198530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84a800ef10 tx=0x7f84a800c3a0 comp rx=0 tx=0).stop 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b5b89700 1 -- 172.21.15.23:0/43606774 reap_dead start 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 shutdown_connections 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 >> 172.21.15.23:0/43606774 conn(0x7f84b0077e10 msgr2=0x7f84b00ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 shutdown_connections 2024-03-16T20:34:40.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:40.323+0000 7f84b75ec700 1 -- 172.21.15.23:0/43606774 wait complete. 2024-03-16T20:34:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:40 smithi031 ceph-mon[36281]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-16T20:34:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:40 smithi189 ceph-mon[36655]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-16T20:34:42.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:42 smithi023 ceph-mon[31879]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T20:34:42.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:42 smithi031 ceph-mon[36281]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T20:34:42.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:42 smithi189 ceph-mon[36655]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-16T20:34:43.887 INFO:teuthology.orchestra.run.smithi023.stdout:163208757300 2024-03-16T20:34:43.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T20:34:44.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:44 smithi023 ceph-mon[31879]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-16T20:34:44.281 INFO:teuthology.orchestra.run.smithi023.stdout:77309411397 2024-03-16T20:34:44.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T20:34:44.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:44 smithi031 ceph-mon[36281]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-16T20:34:44.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:44 smithi189 ceph-mon[36655]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 -- 172.21.15.23:0/1959430870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 msgr2=0x7fac44108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 --2- 172.21.15.23:0/1959430870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 0x7fac44108450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fac40009990 tx=0x7fac40009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 -- 172.21.15.23:0/1959430870 shutdown_connections 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 --2- 172.21.15.23:0/1959430870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 0x7fac44108450 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 --2- 172.21.15.23:0/1959430870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac4410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 --2- 172.21.15.23:0/1959430870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 0x7fac441024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 -- 172.21.15.23:0/1959430870 >> 172.21.15.23:0/1959430870 conn(0x7fac440790b0 msgr2=0x7fac440794c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 -- 172.21.15.23:0/1959430870 shutdown_connections 2024-03-16T20:34:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.488+0000 7fac4b338700 1 -- 172.21.15.23:0/1959430870 wait complete. 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 Processor -- start 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 -- start start 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 0x7fac440742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac44074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 0x7fac44074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac441a6a70 con 0x7fac44108070 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac441a6be0 con 0x7fac44102060 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.489+0000 7fac4b338700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac441a6da0 con 0x7fac44102a00 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac490d4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 0x7fac440742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac44074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac44074820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:52148/0 (socket says 172.21.15.23:52148) 2024-03-16T20:34:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac490d4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 0x7fac440742e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:56676/0 (socket says 172.21.15.23:56676) 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 -- 172.21.15.23:0/572883698 learned_addr learned my addr 172.21.15.23:0/572883698 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac498d5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 0x7fac44074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 -- 172.21.15.23:0/572883698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 msgr2=0x7fac440742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac44102060 0x7fac440742e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 -- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 msgr2=0x7fac44074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac44108070 0x7fac44074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac400095f0 con 0x7fac44102a00 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac488d3700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac44074820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac3400e9a0 tx=0x7fac3400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34009790 con 0x7fac44102a00 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac441a7080 con 0x7fac44102a00 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac34010430 con 0x7fac44102a00 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34018680 con 0x7fac44102a00 2024-03-16T20:34:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.490+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac441a75d0 con 0x7fac44102a00 2024-03-16T20:34:44.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.491+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac44112350 con 0x7fac44102a00 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.492+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fac34010910 con 0x7fac44102a00 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac3a7fc700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac300a1b50 0x7fac300a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac490d4700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac300a1b50 0x7fac300a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fac34014070 con 0x7fac44102a00 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac3a7fc700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] conn(0x7fac300ac810 0x7fac300aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac300af320 con 0x7fac300ac810 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac340ce3d0 con 0x7fac44102a00 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.493+0000 7fac498d5700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] conn(0x7fac300ac810 0x7fac300aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.494+0000 7fac490d4700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac300a1b50 0x7fac300a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b2e0 tx=0x7fac40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.494+0000 7fac498d5700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] conn(0x7fac300ac810 0x7fac300aec50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.495+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== osd.9 v2:172.21.15.23:6826/1125641919 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fac300af320 con 0x7fac300ac810 2024-03-16T20:34:44.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.512+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 --> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac44062670 con 0x7fac300ac810 2024-03-16T20:34:44.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.512+0000 7fac3a7fc700 1 -- 172.21.15.23:0/572883698 <== osd.9 v2:172.21.15.23:6826/1125641919 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fac44062670 con 0x7fac300ac810 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.512+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] conn(0x7fac300ac810 msgr2=0x7fac300aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.512+0000 7fac4b338700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] conn(0x7fac300ac810 0x7fac300aec50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac300a1b50 msgr2=0x7fac300a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fac300a1b50 0x7fac300a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b2e0 tx=0x7fac40005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 msgr2=0x7fac44074820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 --2- 172.21.15.23:0/572883698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac44102a00 0x7fac44074820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac3400e9a0 tx=0x7fac3400ecb0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac498d5700 1 -- 172.21.15.23:0/572883698 reap_dead start 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 shutdown_connections 2024-03-16T20:34:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 >> 172.21.15.23:0/572883698 conn(0x7fac440790b0 msgr2=0x7fac44100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 shutdown_connections 2024-03-16T20:34:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.513+0000 7fac4b338700 1 -- 172.21.15.23:0/572883698 wait complete. 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.705+0000 7fd15be97700 1 -- 172.21.15.23:0/1009728231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154106f20 msgr2=0x7fd154107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.705+0000 7fd15be97700 1 --2- 172.21.15.23:0/1009728231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154106f20 0x7fd154107300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd1500098f0 tx=0x7fd150009c00 comp rx=0 tx=0).stop 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 -- 172.21.15.23:0/1009728231 shutdown_connections 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 --2- 172.21.15.23:0/1009728231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154101880 0x7fd15410dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 --2- 172.21.15.23:0/1009728231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154106f20 0x7fd154107300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 --2- 172.21.15.23:0/1009728231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154100ee0 0x7fd154101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 -- 172.21.15.23:0/1009728231 >> 172.21.15.23:0/1009728231 conn(0x7fd154077e10 msgr2=0x7fd154078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 -- 172.21.15.23:0/1009728231 shutdown_connections 2024-03-16T20:34:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 -- 172.21.15.23:0/1009728231 wait complete. 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 Processor -- start 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.706+0000 7fd15be97700 1 -- start start 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 0x7fd154102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154101880 0x7fd1541030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154106f20 0x7fd154103610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541a6920 con 0x7fd154101880 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541a6a90 con 0x7fd154100ee0 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541a6c50 con 0x7fd154106f20 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159432700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154101880 0x7fd1541030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 0x7fd154102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159432700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154101880 0x7fd1541030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39318/0 (socket says 172.21.15.23:39318) 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159432700 1 -- 172.21.15.23:0/3775539796 learned_addr learned my addr 172.21.15.23:0/3775539796 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 0x7fd154102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:56694/0 (socket says 172.21.15.23:56694) 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15a434700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154106f20 0x7fd154103610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 -- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154106f20 msgr2=0x7fd154103610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd154106f20 0x7fd154103610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 -- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154101880 msgr2=0x7fd1541030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd154101880 0x7fd1541030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440095f0 con 0x7fd154100ee0 2024-03-16T20:34:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd159c33700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 0x7fd154102b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd150000c00 tx=0x7fd15000fa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15001c070 con 0x7fd154100ee0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15000b560 con 0x7fd154100ee0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150003c10 con 0x7fd154100ee0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd150009520 con 0x7fd154100ee0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.707+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1541a7100 con 0x7fd154100ee0 2024-03-16T20:34:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.708+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd154111130 con 0x7fd154100ee0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.709+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd15000ba60 con 0x7fd154100ee0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd14affd700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1400a1b50 0x7fd1400a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd159432700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1400a1b50 0x7fd1400a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd1500ce560 con 0x7fd154100ee0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd14affd700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] conn(0x7fd1400ac810 0x7fd1400aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1400af320 con 0x7fd1400ac810 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1500ce8d0 con 0x7fd154100ee0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd15a434700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] conn(0x7fd1400ac810 0x7fd1400aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.710+0000 7fd159432700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1400a1b50 0x7fd1400a4010 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd1540783a0 tx=0x7fd144009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.711+0000 7fd15a434700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] conn(0x7fd1400ac810 0x7fd1400aec50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.712+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== osd.11 v2:172.21.15.189:6824/702443898 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd1400af320 con 0x7fd1400ac810 2024-03-16T20:34:44.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.730+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 --> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd154105fb0 con 0x7fd1400ac810 2024-03-16T20:34:44.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd14affd700 1 -- 172.21.15.23:0/3775539796 <== osd.11 v2:172.21.15.189:6824/702443898 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd154105fb0 con 0x7fd1400ac810 2024-03-16T20:34:44.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] conn(0x7fd1400ac810 msgr2=0x7fd1400aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] conn(0x7fd1400ac810 0x7fd1400aec50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1400a1b50 msgr2=0x7fd1400a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd1400a1b50 0x7fd1400a4010 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd1540783a0 tx=0x7fd144009310 comp rx=0 tx=0).stop 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 msgr2=0x7fd154102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 --2- 172.21.15.23:0/3775539796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd154100ee0 0x7fd154102b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd150000c00 tx=0x7fd15000fa90 comp rx=0 tx=0).stop 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15a434700 1 -- 172.21.15.23:0/3775539796 reap_dead start 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 shutdown_connections 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.731+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 >> 172.21.15.23:0/3775539796 conn(0x7fd154077e10 msgr2=0x7fd154069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.732+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 shutdown_connections 2024-03-16T20:34:44.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:44.732+0000 7fd15be97700 1 -- 172.21.15.23:0/3775539796 wait complete. 2024-03-16T20:34:45.407 INFO:teuthology.orchestra.run.smithi023.stdout:347892351001 2024-03-16T20:34:45.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-16T20:34:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 -- 172.21.15.23:0/290250879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 msgr2=0x7f02741024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 --2- 172.21.15.23:0/290250879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f02741024b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f02700099c0 tx=0x7f0270009cd0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 -- 172.21.15.23:0/290250879 shutdown_connections 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 --2- 172.21.15.23:0/290250879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 0x7f027410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 --2- 172.21.15.23:0/290250879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 0x7f02741083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 --2- 172.21.15.23:0/290250879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f02741024b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 -- 172.21.15.23:0/290250879 >> 172.21.15.23:0/290250879 conn(0x7f0274079110 msgr2=0x7f0274079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 -- 172.21.15.23:0/290250879 shutdown_connections 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.613+0000 7f027b4c4700 1 -- 172.21.15.23:0/290250879 wait complete. 2024-03-16T20:34:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 Processor -- start 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 -- start start 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f0274198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 0x7f0274198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 0x7f027419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419d3b0 con 0x7f0274102030 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419d520 con 0x7f0274108010 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f027b4c4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419d6e0 con 0x7f02741029f0 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f0279260700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f0274198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f0279260700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f0274198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39328/0 (socket says 172.21.15.23:39328) 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f0279260700 1 -- 172.21.15.23:0/3077292445 learned_addr learned my addr 172.21.15.23:0/3077292445 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f0279a61700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 0x7f027419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.614+0000 7f0278a5f700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 0x7f0274198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 -- 172.21.15.23:0/3077292445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 msgr2=0x7f0274198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 0x7f0274198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 -- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 msgr2=0x7f0274198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f0274198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02700095f0 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279260700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274102030 0x7f0274198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0278a5f700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02741029f0 0x7f0274198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f0279a61700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 0x7f027419cda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f026c00efc0 tx=0x7f026c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c018070 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027419d960 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f026c00f040 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c00cd20 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.615+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02741a5220 con 0x7f0274108010 2024-03-16T20:34:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.616+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02741036c0 con 0x7f0274108010 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.617+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f026c004ad0 con 0x7f0274108010 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f026a7fc700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f02600a1b50 0x7f02600a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f0279260700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f02600a1b50 0x7f02600a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f026c0cdc30 con 0x7f0274108010 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f026a7fc700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] conn(0x7f02600ac810 0x7f02600aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02600af320 con 0x7f02600ac810 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.618+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f026c0ce040 con 0x7f0274108010 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.619+0000 7f0278a5f700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] conn(0x7f02600ac810 0x7f02600aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.619+0000 7f0279260700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f02600a1b50 0x7f02600a4010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0264011550 tx=0x7f0264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.619+0000 7f0278a5f700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] conn(0x7f02600ac810 0x7f02600aec50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.620+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== osd.7 v2:172.21.15.31:6816/2033067958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02600af320 con 0x7f02600ac810 2024-03-16T20:34:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.638+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 --> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0274112320 con 0x7f02600ac810 2024-03-16T20:34:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.638+0000 7f026a7fc700 1 -- 172.21.15.23:0/3077292445 <== osd.7 v2:172.21.15.31:6816/2033067958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0274112320 con 0x7f02600ac810 2024-03-16T20:34:45.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] conn(0x7f02600ac810 msgr2=0x7f02600aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] conn(0x7f02600ac810 0x7f02600aec50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f02600a1b50 msgr2=0x7f02600a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f02600a1b50 0x7f02600a4010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0264011550 tx=0x7f0264009260 comp rx=0 tx=0).stop 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 msgr2=0x7f027419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 --2- 172.21.15.23:0/3077292445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0274108010 0x7f027419cda0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f026c00efc0 tx=0x7f026c00c3a0 comp rx=0 tx=0).stop 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f0279a61700 1 -- 172.21.15.23:0/3077292445 reap_dead start 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 shutdown_connections 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 >> 172.21.15.23:0/3077292445 conn(0x7f0274079110 msgr2=0x7f0274100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 shutdown_connections 2024-03-16T20:34:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:45.639+0000 7f027b4c4700 1 -- 172.21.15.23:0/3077292445 wait complete. 2024-03-16T20:34:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:46 smithi023 ceph-mon[31879]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-16T20:34:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:46 smithi031 ceph-mon[36281]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-16T20:34:46.445 INFO:teuthology.orchestra.run.smithi023.stdout:292057776163 2024-03-16T20:34:46.445 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-16T20:34:46.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:46 smithi189 ceph-mon[36655]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-16T20:34:46.898 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:46.899 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:48.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:48 smithi023 ceph-mon[31879]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-03-16T20:34:48.381 INFO:teuthology.orchestra.run.smithi023.stdout:219043332141 2024-03-16T20:34:48.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T20:34:48.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:48 smithi031 ceph-mon[36281]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-03-16T20:34:48.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:48 smithi189 ceph-mon[36655]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-03-16T20:34:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 -- 172.21.15.23:0/3554419776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 msgr2=0x7f606c06a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 --2- 172.21.15.23:0/3554419776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c06a520 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f605c009990 tx=0x7f605c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 -- 172.21.15.23:0/3554419776 shutdown_connections 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 --2- 172.21.15.23:0/3554419776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c06a520 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 --2- 172.21.15.23:0/3554419776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 0x7f606c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 --2- 172.21.15.23:0/3554419776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c06aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 -- 172.21.15.23:0/3554419776 >> 172.21.15.23:0/3554419776 conn(0x7f606c076f30 msgr2=0x7f606c077340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 -- 172.21.15.23:0/3554419776 shutdown_connections 2024-03-16T20:34:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.808+0000 7f6070e56700 1 -- 172.21.15.23:0/3554419776 wait complete. 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 Processor -- start 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 -- start start 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c1a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 0x7f606c1a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19eee0 con 0x7f606c106fa0 2024-03-16T20:34:48.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19f020 con 0x7f606c06aa60 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6070e56700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19f1e0 con 0x7f606c104a10 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f606ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f606a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c1a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f606ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39344/0 (socket says 172.21.15.23:39344) 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f6069d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 0x7f606c1a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f606ad9d700 1 -- 172.21.15.23:0/3852718532 learned_addr learned my addr 172.21.15.23:0/3852718532 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.809+0000 7f606a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c1a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:56714/0 (socket says 172.21.15.23:56714) 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 -- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 msgr2=0x7f606c1a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 0x7f606c1a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 -- 172.21.15.23:0/3852718532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 msgr2=0x7f606c1a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c1a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064009c40 con 0x7f606c106fa0 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606a59c700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f606c06aa60 0x7f606c1a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f6069d9b700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f606c104a10 0x7f606c1a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f606ad9d700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c19e880 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f606000e940 tx=0x7f606000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60600097b0 con 0x7f606c106fa0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c0095f0 con 0x7f606c106fa0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6060005020 con 0x7f606c106fa0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606000ccf0 con 0x7f606c106fa0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.810+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c1ab590 con 0x7f606c106fa0 2024-03-16T20:34:48.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.811+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f606c1014f0 con 0x7f606c106fa0 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.812+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6060005490 con 0x7f606c106fa0 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f605b7fe700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60540a1b50 0x7f60540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f606a59c700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60540a1b50 0x7f60540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6060014070 con 0x7f606c106fa0 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f605b7fe700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] conn(0x7f60540ac810 0x7f60540aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60540af320 con 0x7f60540ac810 2024-03-16T20:34:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.813+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60600ce230 con 0x7f606c106fa0 2024-03-16T20:34:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.814+0000 7f6069d9b700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] conn(0x7f60540ac810 0x7f60540aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.814+0000 7f606a59c700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60540a1b50 0x7f60540a4010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f606c19f9c0 tx=0x7f605c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.814+0000 7f6069d9b700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] conn(0x7f60540ac810 0x7f60540aec50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.815+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== osd.8 v2:172.21.15.189:6816/3435219550 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f60540af320 con 0x7f60540ac810 2024-03-16T20:34:48.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.831+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 --> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f606c118910 con 0x7f60540ac810 2024-03-16T20:34:48.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.831+0000 7f605b7fe700 1 -- 172.21.15.23:0/3852718532 <== osd.8 v2:172.21.15.189:6816/3435219550 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f606c118910 con 0x7f60540ac810 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.831+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] conn(0x7f60540ac810 msgr2=0x7f60540aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.831+0000 7f6070e56700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] conn(0x7f60540ac810 0x7f60540aec50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60540a1b50 msgr2=0x7f60540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60540a1b50 0x7f60540a4010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f606c19f9c0 tx=0x7f605c005fd0 comp rx=0 tx=0).stop 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 msgr2=0x7f606c19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 --2- 172.21.15.23:0/3852718532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106fa0 0x7f606c19e880 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f606000e940 tx=0x7f606000ec50 comp rx=0 tx=0).stop 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f606ad9d700 1 -- 172.21.15.23:0/3852718532 reap_dead start 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 shutdown_connections 2024-03-16T20:34:48.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 >> 172.21.15.23:0/3852718532 conn(0x7f606c076f30 msgr2=0x7f606c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:48.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 shutdown_connections 2024-03-16T20:34:48.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:48.832+0000 7f6070e56700 1 -- 172.21.15.23:0/3852718532 wait complete. 2024-03-16T20:34:49.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:49 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:34:49.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.628+0000 7f43a0c2e700 1 -- 172.21.15.23:0/2351809533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c06b040 msgr2=0x7f439c10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.628+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/2351809533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c06b040 0x7f439c10ee80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f439400b0a0 tx=0x7f439400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 -- 172.21.15.23:0/2351809533 shutdown_connections 2024-03-16T20:34:49.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/2351809533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c06b040 0x7f439c10ee80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/2351809533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c1080d0 0x7f439c1084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/2351809533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f439c06a680 0x7f439c06ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 -- 172.21.15.23:0/2351809533 >> 172.21.15.23:0/2351809533 conn(0x7f439c077da0 msgr2=0x7f439c0781b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:49.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 -- 172.21.15.23:0/2351809533 shutdown_connections 2024-03-16T20:34:49.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 -- 172.21.15.23:0/2351809533 wait complete. 2024-03-16T20:34:49.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 Processor -- start 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.629+0000 7f43a0c2e700 1 -- start start 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c06a680 0x7f439c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f439c06b040 0x7f439c198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 0x7f439c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c19d500 con 0x7f439c1080d0 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c19d670 con 0x7f439c06b040 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f43a0c2e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c19d830 con 0x7f439c06a680 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 0x7f439c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f4399d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f439c06b040 0x7f439c198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c06a680 0x7f439c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 0x7f439c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39372/0 (socket says 172.21.15.23:39372) 2024-03-16T20:34:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439ad9d700 1 -- 172.21.15.23:0/3918095230 learned_addr learned my addr 172.21.15.23:0/3918095230 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 -- 172.21.15.23:0/3918095230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f439c06b040 msgr2=0x7f439c198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f439c06b040 0x7f439c198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 -- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 msgr2=0x7f439c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 0x7f439c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439a59c700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394009d00 con 0x7f439c06a680 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.630+0000 7f439ad9d700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c1080d0 0x7f439c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f439a59c700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c06a680 0x7f439c198620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f438400e9f0 tx=0x7f438400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400c9c0 con 0x7f439c06a680 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c19db10 con 0x7f439c06a680 2024-03-16T20:34:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4384010430 con 0x7f439c06a680 2024-03-16T20:34:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384018660 con 0x7f439c06a680 2024-03-16T20:34:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.631+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c1a5370 con 0x7f439c06a680 2024-03-16T20:34:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.632+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f439c1123e0 con 0x7f439c06a680 2024-03-16T20:34:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.633+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f4384010910 con 0x7f439c06a680 2024-03-16T20:34:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f43937fe700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f43880a1b50 0x7f43880a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f4399d9b700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f43880a1b50 0x7f43880a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4384014070 con 0x7f439c06a680 2024-03-16T20:34:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f43937fe700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] conn(0x7f43880ac810 0x7f43880aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43880af320 con 0x7f43880ac810 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43840ce610 con 0x7f439c06a680 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f439ad9d700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] conn(0x7f43880ac810 0x7f43880aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f4399d9b700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f43880a1b50 0x7f43880a4010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f439c078350 tx=0x7f438c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.634+0000 7f439ad9d700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] conn(0x7f43880ac810 0x7f43880aec50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.635+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== osd.6 v2:172.21.15.23:6818/103652673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f43880af320 con 0x7f43880ac810 2024-03-16T20:34:49.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.652+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 --> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f439c0626a0 con 0x7f43880ac810 2024-03-16T20:34:49.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43937fe700 1 -- 172.21.15.23:0/3918095230 <== osd.6 v2:172.21.15.23:6818/103652673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f439c0626a0 con 0x7f43880ac810 2024-03-16T20:34:49.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] conn(0x7f43880ac810 msgr2=0x7f43880aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] conn(0x7f43880ac810 0x7f43880aec50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f43880a1b50 msgr2=0x7f43880a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f43880a1b50 0x7f43880a4010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f439c078350 tx=0x7f438c0073d0 comp rx=0 tx=0).stop 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c06a680 msgr2=0x7f439c198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 --2- 172.21.15.23:0/3918095230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f439c06a680 0x7f439c198620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f438400e9f0 tx=0x7f438400ed00 comp rx=0 tx=0).stop 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f439ad9d700 1 -- 172.21.15.23:0/3918095230 reap_dead start 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 shutdown_connections 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 >> 172.21.15.23:0/3918095230 conn(0x7f439c077da0 msgr2=0x7f439c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 shutdown_connections 2024-03-16T20:34:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:49.653+0000 7f43a0c2e700 1 -- 172.21.15.23:0/3918095230 wait complete. 2024-03-16T20:34:49.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:49 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:34:49.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:49 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:34:50.265 INFO:teuthology.orchestra.run.smithi023.stdout:55834574922 2024-03-16T20:34:50.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T20:34:50.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:50 smithi023 ceph-mon[31879]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.666 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-03-16T20:34:50.667 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-16T20:34:50.667 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.667 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.667 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:50 smithi189 ceph-mon[36655]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-16T20:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:50.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-16T20:34:50.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:50 smithi031 ceph-mon[36281]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-16T20:34:51.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.591 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.592 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-16T20:34:51.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T20:34:51.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.690 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-16T20:34:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-16T20:34:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-16T20:34:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-16T20:34:51.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 10]}]': finished 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: osdmap e91: 12 total, 12 up, 12 in 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.722 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:34:51.958 INFO:teuthology.orchestra.run.smithi023.stdout:257698037801 2024-03-16T20:34:51.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-16T20:34:52.435 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:52 smithi023 ceph-mon[31879]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T20:34:52.435 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:52 smithi023 ceph-mon[31879]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T20:34:52.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:52 smithi031 ceph-mon[36281]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T20:34:52.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:52 smithi031 ceph-mon[36281]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T20:34:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:52 smithi189 ceph-mon[36655]: osdmap e92: 12 total, 12 up, 12 in 2024-03-16T20:34:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:52 smithi189 ceph-mon[36655]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 -- 172.21.15.23:0/3995584195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102a20 msgr2=0x7f1dec10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/3995584195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102a20 0x7f1dec10ee80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b0a0 tx=0x7f1de400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 -- 172.21.15.23:0/3995584195 shutdown_connections 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/3995584195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102a20 0x7f1dec10ee80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/3995584195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec108140 0x7f1dec108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/3995584195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec102060 0x7f1dec1024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 -- 172.21.15.23:0/3995584195 >> 172.21.15.23:0/3995584195 conn(0x7f1dec0790a0 msgr2=0x7f1dec0794b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 -- 172.21.15.23:0/3995584195 shutdown_connections 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.391+0000 7f1df1ac2700 1 -- 172.21.15.23:0/3995584195 wait complete. 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.392+0000 7f1df1ac2700 1 Processor -- start 2024-03-16T20:34:52.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.392+0000 7f1df1ac2700 1 -- start start 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 0x7f1dec1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec102a20 0x7f1dec198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec108140 0x7f1dec19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19d4c0 con 0x7f1dec102060 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19d630 con 0x7f1dec102a20 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1df1ac2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19d7f0 con 0x7f1dec108140 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 0x7f1dec1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 0x7f1dec1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47070/0 (socket says 172.21.15.23:47070) 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 -- 172.21.15.23:0/4194236578 learned_addr learned my addr 172.21.15.23:0/4194236578 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:52.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deaffd700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec102a20 0x7f1dec198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1debfff700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec108140 0x7f1dec19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 -- 172.21.15.23:0/4194236578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec108140 msgr2=0x7f1dec19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec108140 0x7f1dec19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 -- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec102a20 msgr2=0x7f1dec198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec102a20 0x7f1dec198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4009d00 con 0x7f1dec102060 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deaffd700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec102a20 0x7f1dec198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.393+0000 7f1deb7fe700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 0x7f1dec1985e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00e9a0 tx=0x7f1ddc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.394+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc009790 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.394+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc010430 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.394+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc018680 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.394+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec19dad0 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.394+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a5330 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.395+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1dec103720 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.396+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1ddc010910 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1de8ff9700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd40a1b50 0x7f1dd40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f1ddc014070 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1deaffd700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd40a1b50 0x7f1dd40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1de8ff9700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] conn(0x7f1dd40acc30 0x7f1dd40af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dd40af7a0 con 0x7f1dd40acc30 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ddc0ce4c0 con 0x7f1dec102060 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1debfff700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] conn(0x7f1dd40acc30 0x7f1dd40af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1deaffd700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd40a1b50 0x7f1dd40a4010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0795c0 tx=0x7f1de0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.397+0000 7f1debfff700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] conn(0x7f1dd40acc30 0x7f1dd40af0d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.399+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== osd.4 v2:172.21.15.31:6808/2047837401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1dd40af7a0 con 0x7f1dd40acc30 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.416+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 --> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1dec112340 con 0x7f1dd40acc30 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.416+0000 7f1de8ff9700 1 -- 172.21.15.23:0/4194236578 <== osd.4 v2:172.21.15.31:6808/2047837401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1dec112340 con 0x7f1dd40acc30 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] conn(0x7f1dd40acc30 msgr2=0x7f1dd40af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] conn(0x7f1dd40acc30 0x7f1dd40af0d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd40a1b50 msgr2=0x7f1dd40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1dd40a1b50 0x7f1dd40a4010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0795c0 tx=0x7f1de0009260 comp rx=0 tx=0).stop 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 msgr2=0x7f1dec1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 --2- 172.21.15.23:0/4194236578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dec102060 0x7f1dec1985e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00e9a0 tx=0x7f1ddc00ecb0 comp rx=0 tx=0).stop 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1debfff700 1 -- 172.21.15.23:0/4194236578 reap_dead start 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 shutdown_connections 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 >> 172.21.15.23:0/4194236578 conn(0x7f1dec0790a0 msgr2=0x7f1dec100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 shutdown_connections 2024-03-16T20:34:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:52.417+0000 7f1df1ac2700 1 -- 172.21.15.23:0/4194236578 wait complete. 2024-03-16T20:34:53.499 INFO:teuthology.orchestra.run.smithi023.stdout:197568495667 2024-03-16T20:34:53.499 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T20:34:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:53 smithi031 ceph-mon[36281]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-16T20:34:53.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:53 smithi189 ceph-mon[36655]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-16T20:34:53.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:53 smithi023 ceph-mon[31879]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-16T20:34:54.441 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 -- 172.21.15.23:0/135086902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 msgr2=0x7f71d806a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 --2- 172.21.15.23:0/135086902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d806a0f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f71c80099c0 tx=0x7f71c8009cd0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 -- 172.21.15.23:0/135086902 shutdown_connections 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 --2- 172.21.15.23:0/135086902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71d806a630 0x7f71d810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 --2- 172.21.15.23:0/135086902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71d8108030 0x7f71d8108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 --2- 172.21.15.23:0/135086902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d806a0f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.538+0000 7f71df911700 1 -- 172.21.15.23:0/135086902 >> 172.21.15.23:0/135086902 conn(0x7f71d8077e10 msgr2=0x7f71d8078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:54.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- 172.21.15.23:0/135086902 shutdown_connections 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- 172.21.15.23:0/135086902 wait complete. 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 Processor -- start 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- start start 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71d806a630 0x7f71d81989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71d8108030 0x7f71d819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d819d380 con 0x7f71d8069c90 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d819d4f0 con 0x7f71d8108030 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71df911700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d819d6b0 con 0x7f71d806a630 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71dd6ad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71dd6ad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d81984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47078/0 (socket says 172.21.15.23:47078) 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.539+0000 7f71dd6ad700 1 -- 172.21.15.23:0/2764175863 learned_addr learned my addr 172.21.15.23:0/2764175863 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:54.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71ddeae700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71d8108030 0x7f71d819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dceac700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71d806a630 0x7f71d81989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 -- 172.21.15.23:0/2764175863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71d806a630 msgr2=0x7f71d81989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71d806a630 0x7f71d81989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 -- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71d8108030 msgr2=0x7f71d819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71d8108030 0x7f71d819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c80095f0 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71dd6ad700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d81984a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f71d400e9a0 tx=0x7f71d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4009790 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71d4010430 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d819d990 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4018680 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.540+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d81a51f0 con 0x7f71d8069c90 2024-03-16T20:34:54.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.541+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71d81036e0 con 0x7f71d8069c90 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.542+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f71d4010910 con 0x7f71d8069c90 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ce7fc700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71c40a1b50 0x7f71c40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71dceac700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71c40a1b50 0x7f71c40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f71d4014070 con 0x7f71d8069c90 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ce7fc700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] conn(0x7f71c40acc30 0x7f71c40af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71c40af7a0 con 0x7f71c40acc30 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71d40ce4c0 con 0x7f71d8069c90 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71ddeae700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] conn(0x7f71c40acc30 0x7f71c40af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.543+0000 7f71dceac700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71c40a1b50 0x7f71c40a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:54.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.544+0000 7f71ddeae700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] conn(0x7f71c40acc30 0x7f71c40af0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:54.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.545+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== osd.10 v2:172.21.15.31:6824/3304713388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f71c40af7a0 con 0x7f71c40acc30 2024-03-16T20:34:54.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.564+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 --> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71d8112320 con 0x7f71c40acc30 2024-03-16T20:34:54.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.564+0000 7f71ce7fc700 1 -- 172.21.15.23:0/2764175863 <== osd.10 v2:172.21.15.31:6824/3304713388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f71d8112320 con 0x7f71c40acc30 2024-03-16T20:34:54.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] conn(0x7f71c40acc30 msgr2=0x7f71c40af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] conn(0x7f71c40acc30 0x7f71c40af0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71c40a1b50 msgr2=0x7f71c40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f71c40a1b50 0x7f71c40a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c8005fb0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 msgr2=0x7f71d81984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 --2- 172.21.15.23:0/2764175863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71d8069c90 0x7f71d81984a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f71d400e9a0 tx=0x7f71d400ecb0 comp rx=0 tx=0).stop 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71ddeae700 1 -- 172.21.15.23:0/2764175863 reap_dead start 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 shutdown_connections 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 >> 172.21.15.23:0/2764175863 conn(0x7f71d8077e10 msgr2=0x7f71d80ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 shutdown_connections 2024-03-16T20:34:54.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:54.565+0000 7f71df911700 1 -- 172.21.15.23:0/2764175863 wait complete. 2024-03-16T20:34:54.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:54 smithi023 ceph-mon[31879]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:54.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:54 smithi189 ceph-mon[36655]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:54.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:54 smithi031 ceph-mon[36281]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:55.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 -- 172.21.15.23:0/2339111121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 msgr2=0x7f1f6c105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 --2- 172.21.15.23:0/2339111121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c105b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009990 tx=0x7f1f5c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 -- 172.21.15.23:0/2339111121 shutdown_connections 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 --2- 172.21.15.23:0/2339111121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c106200 0x7f1f6c10ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 --2- 172.21.15.23:0/2339111121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c101170 0x7f1f6c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 --2- 172.21.15.23:0/2339111121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c105b20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 -- 172.21.15.23:0/2339111121 >> 172.21.15.23:0/2339111121 conn(0x7f1f6c076f00 msgr2=0x7f1f6c077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 -- 172.21.15.23:0/2339111121 shutdown_connections 2024-03-16T20:34:55.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.475+0000 7f1f74156700 1 -- 172.21.15.23:0/2339111121 wait complete. 2024-03-16T20:34:55.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.476+0000 7f1f74156700 1 Processor -- start 2024-03-16T20:34:55.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.476+0000 7f1f74156700 1 -- start start 2024-03-16T20:34:55.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 0x7f1f6c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c106200 0x7f1f6c1a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c19a1c0 con 0x7f1f6c101b20 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c1a49d0 con 0x7f1f6c101170 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c1a4b10 con 0x7f1f6c106200 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f71ef2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 0x7f1f6c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47094/0 (socket says 172.21.15.23:47094) 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 -- 172.21.15.23:0/4126458152 learned_addr learned my addr 172.21.15.23:0/4126458152 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f726f3700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c106200 0x7f1f6c1a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 -- 172.21.15.23:0/4126458152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c106200 msgr2=0x7f1f6c1a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c106200 0x7f1f6c1a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 -- 172.21.15.23:0/4126458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 msgr2=0x7f1f6c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:55.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 0x7f1f6c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 -- 172.21.15.23:0/4126458152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c0095f0 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f71ef2700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 0x7f1f6c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f716f1700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c004730 tx=0x7f1f5c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c01d070 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f5c00b610 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c1a4d40 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.477+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c0173f0 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.478+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c1a51b0 con 0x7f1f6c101b20 2024-03-16T20:34:55.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.478+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c1094b0 con 0x7f1f6c101b20 2024-03-16T20:34:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.479+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1f5c00fc60 con 0x7f1f6c101b20 2024-03-16T20:34:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.480+0000 7f1f62ffd700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 0x7f1f580a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.480+0000 7f1f71ef2700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 0x7f1f580a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.480+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f1f5c0cf280 con 0x7f1f6c101b20 2024-03-16T20:34:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.481+0000 7f1f71ef2700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 0x7f1f580a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f68011490 tx=0x7f1f68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:55.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.484+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1f5c098020 con 0x7f1f6c101b20 2024-03-16T20:34:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.687+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1f6c06aca0 con 0x7f1f6c101b20 2024-03-16T20:34:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.688+0000 7f1f62ffd700 1 -- 172.21.15.23:0/4126458152 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f5c09b6e0 con 0x7f1f6c101b20 2024-03-16T20:34:55.693 INFO:teuthology.orchestra.run.smithi023.stdout:77309411400 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 msgr2=0x7f1f580a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 0x7f1f580a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1f68011490 tx=0x7f1f68009260 comp rx=0 tx=0).stop 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 msgr2=0x7f1f6c199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c004730 tx=0x7f1f5c004760 comp rx=0 tx=0).stop 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 shutdown_connections 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f580a1bd0 0x7f1f580a4090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f6c106200 0x7f1f6c1a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f6c101170 0x7f1f6c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 --2- 172.21.15.23:0/4126458152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f6c101b20 0x7f1f6c199950 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 >> 172.21.15.23:0/4126458152 conn(0x7f1f6c076f00 msgr2=0x7f1f6c1006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 shutdown_connections 2024-03-16T20:34:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:55.690+0000 7f1f74156700 1 -- 172.21.15.23:0/4126458152 wait complete. 2024-03-16T20:34:56.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4126458152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T20:34:56.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:55 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4126458152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T20:34:56.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:55 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4126458152' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T20:34:57.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:56 smithi023 ceph-mon[31879]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:57.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/972734909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T20:34:57.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:56 smithi031 ceph-mon[36281]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:57.189 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/972734909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T20:34:57.208 INFO:teuthology.orchestra.run.smithi023.stdout:313532612639 2024-03-16T20:34:57.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-16T20:34:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:56 smithi189 ceph-mon[36655]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:57.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/972734909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T20:34:57.748 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411397 got 77309411400 for osd.2 2024-03-16T20:34:57.748 DEBUG:teuthology.parallel:result is None 2024-03-16T20:34:57.876 INFO:teuthology.orchestra.run.smithi023.stdout:141733920830 2024-03-16T20:34:57.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T20:34:58.301 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:58 smithi023 ceph-mon[31879]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:58.301 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:58 smithi023 ceph-mon[31879]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-16T20:34:58.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:34:58 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:34:58.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:58 smithi031 ceph-mon[36281]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:58.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:58 smithi031 ceph-mon[36281]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-16T20:34:58.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:34:58 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:34:58.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:58 smithi189 ceph-mon[36655]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:34:58.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:58 smithi189 ceph-mon[36655]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-16T20:34:58.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:34:58 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 -- 172.21.15.23:0/2013115366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f406a630 msgr2=0x7f39f410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 --2- 172.21.15.23:0/2013115366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f406a630 0x7f39f410ee70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f39ec009fc0 tx=0x7f39ec00b330 comp rx=0 tx=0).stop 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 -- 172.21.15.23:0/2013115366 shutdown_connections 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 --2- 172.21.15.23:0/2013115366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f406a630 0x7f39f410ee70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 --2- 172.21.15.23:0/2013115366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4108030 0x7f39f4108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 --2- 172.21.15.23:0/2013115366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f4069c90 0x7f39f406a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 -- 172.21.15.23:0/2013115366 >> 172.21.15.23:0/2013115366 conn(0x7f39f4077e10 msgr2=0x7f39f4078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 -- 172.21.15.23:0/2013115366 shutdown_connections 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.480+0000 7f39fb7af700 1 -- 172.21.15.23:0/2013115366 wait complete. 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 Processor -- start 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 -- start start 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f406a630 0x7f39f41014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 0x7f39f4101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f41a6a80 con 0x7f39f406a630 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f41a6bf0 con 0x7f39f4069c90 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39fb7af700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f41a6db0 con 0x7f39f4108030 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39f954b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39f8d4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f406a630 0x7f39f41014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39f954b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:42748/0 (socket says 172.21.15.23:42748) 2024-03-16T20:34:58.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39f954b700 1 -- 172.21.15.23:0/972734909 learned_addr learned my addr 172.21.15.23:0/972734909 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:34:58.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.481+0000 7f39f9d4c700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 0x7f39f4101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:58.513 INFO:teuthology.orchestra.run.smithi023.stdout:107374182467 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 -- 172.21.15.23:0/972734909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 msgr2=0x7f39f4101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 0x7f39f4101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 -- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f406a630 msgr2=0x7f39f41014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f406a630 0x7f39f41014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 -- 172.21.15.23:0/972734909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec009c40 con 0x7f39f4069c90 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f9d4c700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 0x7f39f4101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39f954b700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39f000e9c0 tx=0x7f39f000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0004a30 con 0x7f39f4069c90 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f41a7090 con 0x7f39f4069c90 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39f000cca0 con 0x7f39f4069c90 2024-03-16T20:34:58.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0005580 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.482+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f41a75e0 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.483+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f410a550 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.484+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f39f0004c20 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.485+0000 7f39ea7fc700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 0x7f39e00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.485+0000 7f39f8d4a700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 0x7f39e00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.485+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f39f0014070 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.485+0000 7f39f8d4a700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 0x7f39e00a4010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39f40783f0 tx=0x7f39e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.494+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f39f00968b0 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.708+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f39f4068ca0 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.709+0000 7f39ea7fc700 1 -- 172.21.15.23:0/972734909 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f39f0099f70 con 0x7f39f4069c90 2024-03-16T20:34:58.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 msgr2=0x7f39e00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 0x7f39e00a4010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39f40783f0 tx=0x7f39e4009260 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 msgr2=0x7f39f4100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39f000e9c0 tx=0x7f39f000ecd0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 shutdown_connections 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39e00a1b50 0x7f39e00a4010 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39f4108030 0x7f39f4101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39f4069c90 0x7f39f4100fb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 --2- 172.21.15.23:0/972734909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39f406a630 0x7f39f41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 >> 172.21.15.23:0/972734909 conn(0x7f39f4077e10 msgr2=0x7f39f40ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 shutdown_connections 2024-03-16T20:34:58.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:34:56.711+0000 7f39fb7af700 1 -- 172.21.15.23:0/972734909 wait complete. 2024-03-16T20:34:59.142 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182467 for osd.3 2024-03-16T20:34:59.142 DEBUG:teuthology.parallel:result is None 2024-03-16T20:34:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:00.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:00 smithi023 ceph-mon[31879]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:00 smithi031 ceph-mon[36281]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:00.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 -- 172.21.15.23:0/1832048068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc102a20 msgr2=0x7f42dc10ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/1832048068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc102a20 0x7f42dc10ee80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f42d400b0a0 tx=0x7f42d400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 -- 172.21.15.23:0/1832048068 shutdown_connections 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/1832048068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc102a20 0x7f42dc10ee80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/1832048068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc108140 0x7f42dc108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/1832048068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102060 0x7f42dc1024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 -- 172.21.15.23:0/1832048068 >> 172.21.15.23:0/1832048068 conn(0x7f42dc0790a0 msgr2=0x7f42dc0794b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 -- 172.21.15.23:0/1832048068 shutdown_connections 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.457+0000 7f42e1bf6700 1 -- 172.21.15.23:0/1832048068 wait complete. 2024-03-16T20:35:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 Processor -- start 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 -- start start 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 0x7f42dc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 0x7f42dc19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc19d580 con 0x7f42dc108140 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc19d6f0 con 0x7f42dc102060 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42e1bf6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc19d8b0 con 0x7f42dc102a20 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42db7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42daffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 0x7f42dc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.458+0000 7f42dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 0x7f42dc19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42db7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:42772/0 (socket says 172.21.15.23:42772) 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42db7fe700 1 -- 172.21.15.23:0/2404008380 learned_addr learned my addr 172.21.15.23:0/2404008380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 -- 172.21.15.23:0/2404008380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 msgr2=0x7f42dc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 -- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 msgr2=0x7f42dc19cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 0x7f42dc19cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 -- 172.21.15.23:0/2404008380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4009d00 con 0x7f42dc102a20 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42dbfff700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 0x7f42dc19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42db7fe700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42daffd700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 0x7f42dc198c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e9f0 tx=0x7f42d000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d000c9c0 con 0x7f42dc102a20 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc19db90 con 0x7f42dc102a20 2024-03-16T20:35:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42d0010430 con 0x7f42dc102a20 2024-03-16T20:35:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0018660 con 0x7f42dc102a20 2024-03-16T20:35:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.459+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc1a53f0 con 0x7f42dc102a20 2024-03-16T20:35:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.460+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc10a5d0 con 0x7f42dc102a20 2024-03-16T20:35:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.461+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f42d0010910 con 0x7f42dc102a20 2024-03-16T20:35:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.462+0000 7f42d8ff9700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 0x7f42c40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.462+0000 7f42db7fe700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 0x7f42c40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.462+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f42d0014070 con 0x7f42dc102a20 2024-03-16T20:35:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.462+0000 7f42db7fe700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 0x7f42c40a4010 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42cc011550 tx=0x7f42cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:00.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.466+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f42d0096750 con 0x7f42dc102a20 2024-03-16T20:35:00.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:00 smithi189 ceph-mon[36655]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:00.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.673+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f42dc199a00 con 0x7f42dc102a20 2024-03-16T20:35:00.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.674+0000 7f42d8ff9700 1 -- 172.21.15.23:0/2404008380 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42d00105a0 con 0x7f42dc102a20 2024-03-16T20:35:00.679 INFO:teuthology.orchestra.run.smithi023.stdout:34359738451 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 msgr2=0x7f42c40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 0x7f42c40a4010 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42cc011550 tx=0x7f42cc009260 comp rx=0 tx=0).stop 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 msgr2=0x7f42dc198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 0x7f42dc198c30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e9f0 tx=0x7f42d000ed00 comp rx=0 tx=0).stop 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 shutdown_connections 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f42c40a1b50 0x7f42c40a4010 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42dc108140 0x7f42dc19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42dc102060 0x7f42dc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 --2- 172.21.15.23:0/2404008380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42dc102a20 0x7f42dc198c30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:00.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 >> 172.21.15.23:0/2404008380 conn(0x7f42dc0790a0 msgr2=0x7f42dc100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:00.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 shutdown_connections 2024-03-16T20:35:00.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:00.676+0000 7f42e1bf6700 1 -- 172.21.15.23:0/2404008380 wait complete. 2024-03-16T20:35:01.141 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:01.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2404008380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T20:35:01.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2404008380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T20:35:01.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2404008380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T20:35:01.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738451 for osd.0 2024-03-16T20:35:01.586 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:01.750 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:01.751 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 -- 172.21.15.23:0/3509505990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4106260 msgr2=0x7efca410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 --2- 172.21.15.23:0/3509505990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4106260 0x7efca410de30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b0a0 tx=0x7efc9c00b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 -- 172.21.15.23:0/3509505990 shutdown_connections 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 --2- 172.21.15.23:0/3509505990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca4101940 0x7efca4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 --2- 172.21.15.23:0/3509505990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4106260 0x7efca410de30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 --2- 172.21.15.23:0/3509505990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca41022f0 0x7efca4105b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 -- 172.21.15.23:0/3509505990 >> 172.21.15.23:0/3509505990 conn(0x7efca40774e0 msgr2=0x7efca40fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:01.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 -- 172.21.15.23:0/3509505990 shutdown_connections 2024-03-16T20:35:01.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.753+0000 7efcab5b4700 1 -- 172.21.15.23:0/3509505990 wait complete. 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.754+0000 7efcab5b4700 1 Processor -- start 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.754+0000 7efcab5b4700 1 -- start start 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.754+0000 7efcab5b4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 0x7efca4199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.754+0000 7efcab5b4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca41022f0 0x7efca4199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.754+0000 7efcab5b4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efcab5b4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca419a2e0 con 0x7efca41022f0 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efcab5b4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca41a4ac0 con 0x7efca4101940 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:36172/0 (socket says 172.21.15.23:36172) 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 -- 172.21.15.23:0/1591015987 learned_addr learned my addr 172.21.15.23:0/1591015987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca41a4c00 con 0x7efca4106260 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca8b4f700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca41022f0 0x7efca4199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9350700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 0x7efca4199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 -- 172.21.15.23:0/1591015987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 msgr2=0x7efca4199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 0x7efca4199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 -- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca41022f0 msgr2=0x7efca4199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca41022f0 0x7efca4199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9b51700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c009d00 con 0x7efca4106260 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.755+0000 7efca9350700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 0x7efca4199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efca9b51700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b070 tx=0x7efc9c00bdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00e040 con 0x7efca4106260 2024-03-16T20:35:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc9c01d470 con 0x7efca4106260 2024-03-16T20:35:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c005280 con 0x7efca4106260 2024-03-16T20:35:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca41a4e50 con 0x7efca4106260 2024-03-16T20:35:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.756+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca41a5270 con 0x7efca4106260 2024-03-16T20:35:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.757+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca4109580 con 0x7efca4106260 2024-03-16T20:35:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.758+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7efc9c01d5e0 con 0x7efca4106260 2024-03-16T20:35:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.759+0000 7efc9a7fc700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 0x7efc900a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.759+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7efc9c0cf300 con 0x7efca4106260 2024-03-16T20:35:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.759+0000 7efca9350700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 0x7efc900a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.760+0000 7efca9350700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 0x7efc900a4010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efca0011490 tx=0x7efca0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.762+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efc9c098100 con 0x7efca4106260 2024-03-16T20:35:01.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.985+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7efca4068ca0 con 0x7efca4106260 2024-03-16T20:35:01.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.986+0000 7efc9a7fc700 1 -- 172.21.15.23:0/1591015987 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efc9c09b7c0 con 0x7efca4106260 2024-03-16T20:35:01.994 INFO:teuthology.orchestra.run.smithi023.stdout:292057776166 2024-03-16T20:35:01.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 msgr2=0x7efc900a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:01.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 0x7efc900a4010 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efca0011490 tx=0x7efca0009260 comp rx=0 tx=0).stop 2024-03-16T20:35:01.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 msgr2=0x7efca41a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b070 tx=0x7efc9c00bdd0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 shutdown_connections 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efc900a1b50 0x7efc900a4010 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efca4106260 0x7efca41a4580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca4101940 0x7efca4199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 --2- 172.21.15.23:0/1591015987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca41022f0 0x7efca4199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 >> 172.21.15.23:0/1591015987 conn(0x7efca40774e0 msgr2=0x7efca4104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 shutdown_connections 2024-03-16T20:35:01.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:01.988+0000 7efcab5b4700 1 -- 172.21.15.23:0/1591015987 wait complete. 2024-03-16T20:35:02.246 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:02 smithi023 ceph-mon[31879]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:02.247 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1591015987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T20:35:02.339 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:02 smithi031 ceph-mon[36281]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:02.339 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1591015987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T20:35:02.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:02 smithi189 ceph-mon[36655]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:02.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1591015987' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-16T20:35:04.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776166 for osd.9 2024-03-16T20:35:04.217 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:04.294 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:04 smithi023 ceph-mon[31879]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:04.441 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:04 smithi031 ceph-mon[36281]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:04.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:04 smithi189 ceph-mon[36655]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-16T20:35:05.029 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:05.029 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:06.012 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:06.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/255171013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- 172.21.15.23:0/499410569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 msgr2=0x7fa38c10ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 --2- 172.21.15.23:0/499410569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c10ef10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b0a0 tx=0x7fa38400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- 172.21.15.23:0/499410569 shutdown_connections 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 --2- 172.21.15.23:0/499410569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c10ef10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 --2- 172.21.15.23:0/499410569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c100b70 0x7fa38c100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 --2- 172.21.15.23:0/499410569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c101520 0x7fa38c106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- 172.21.15.23:0/499410569 >> 172.21.15.23:0/499410569 conn(0x7fa38c076210 msgr2=0x7fa38c076620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- 172.21.15.23:0/499410569 shutdown_connections 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- 172.21.15.23:0/499410569 wait complete. 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 Processor -- start 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.578+0000 7fa391e24700 1 -- start start 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c100b70 0x7fa38c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19d4f0 con 0x7fa38c107390 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19d660 con 0x7fa38c101520 2024-03-16T20:35:06.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa391e24700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c19d820 con 0x7fa38c100b70 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c100b70 0x7fa38c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55710/0 (socket says 172.21.15.23:55710) 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:55098/0 (socket says 172.21.15.23:55098) 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 -- 172.21.15.23:0/255171013 learned_addr learned my addr 172.21.15.23:0/255171013 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 -- 172.21.15.23:0/255171013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c100b70 msgr2=0x7fa38c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c100b70 0x7fa38c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 -- 172.21.15.23:0/255171013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 msgr2=0x7fa38c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 -- 172.21.15.23:0/255171013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa384009d00 con 0x7fa38c107390 2024-03-16T20:35:06.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38affd700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stdout:55834574927 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa38bfff700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b070 tx=0x7fa38400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.579+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400e040 con 0x7fa38c107390 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.580+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c19daa0 con 0x7fa38c107390 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.580+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa38401d470 con 0x7fa38c107390 2024-03-16T20:35:06.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.580+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384005260 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.580+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c1a5360 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.580+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c10a5d0 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.582+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa38401d5e0 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.582+0000 7fa388ff9700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 0x7fa3740a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.582+0000 7fa38b7fe700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 0x7fa3740a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.582+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fa3840cf330 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.583+0000 7fa38b7fe700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 0x7fa3740a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009c60 tx=0x7fa37c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.586+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3840981d0 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.791+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa38c199970 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.791+0000 7fa388ff9700 1 -- 172.21.15.23:0/255171013 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa38401ae60 con 0x7fa38c107390 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 msgr2=0x7fa3740a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 0x7fa3740a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009c60 tx=0x7fa37c009260 comp rx=0 tx=0).stop 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 msgr2=0x7fa38c19cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:06.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b070 tx=0x7fa38400bdd0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 shutdown_connections 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fa3740a1b50 0x7fa3740a4010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa38c107390 0x7fa38c19cee0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa38c100b70 0x7fa38c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.793+0000 7fa391e24700 1 --2- 172.21.15.23:0/255171013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa38c101520 0x7fa38c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.794+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 >> 172.21.15.23:0/255171013 conn(0x7fa38c076210 msgr2=0x7fa38c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.794+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 shutdown_connections 2024-03-16T20:35:06.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:05.794+0000 7fa391e24700 1 -- 172.21.15.23:0/255171013 wait complete. 2024-03-16T20:35:06.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/255171013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T20:35:06.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/255171013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T20:35:06.715 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574927 for osd.1 2024-03-16T20:35:06.715 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:07.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:06 smithi023 ceph-mon[31879]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:07.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:06 smithi031 ceph-mon[36281]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:07.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:06 smithi189 ceph-mon[36655]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:08.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:08 smithi023 ceph-mon[31879]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-16T20:35:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:08 smithi031 ceph-mon[36281]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-16T20:35:08.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:08 smithi189 ceph-mon[36655]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 -- 172.21.15.23:0/7311850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406a780 msgr2=0x7f93b406ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 --2- 172.21.15.23:0/7311850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406a780 0x7f93b406ab60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009990 tx=0x7f93a4009ca0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 -- 172.21.15.23:0/7311850 shutdown_connections 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 --2- 172.21.15.23:0/7311850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406a780 0x7f93b406ab60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 --2- 172.21.15.23:0/7311850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b4107360 0x7f93b410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 --2- 172.21.15.23:0/7311850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b406b130 0x7f93b4106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 -- 172.21.15.23:0/7311850 >> 172.21.15.23:0/7311850 conn(0x7f93b4076f30 msgr2=0x7f93b4077340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:09.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 -- 172.21.15.23:0/7311850 shutdown_connections 2024-03-16T20:35:09.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.101+0000 7f93b9313700 1 -- 172.21.15.23:0/7311850 wait complete. 2024-03-16T20:35:09.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 Processor -- start 2024-03-16T20:35:09.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 -- start start 2024-03-16T20:35:09.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 0x7f93b41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:09.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4107360 0x7f93b419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d480 con 0x7f93b406b130 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d5f0 con 0x7f93b406a780 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b9313700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d7b0 con 0x7f93b4107360 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.102+0000 7f93b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55732/0 (socket says 172.21.15.23:55732) 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 0x7f93b41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 -- 172.21.15.23:0/422232319 learned_addr learned my addr 172.21.15.23:0/422232319 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b37fe700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4107360 0x7f93b419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 -- 172.21.15.23:0/422232319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4107360 msgr2=0x7f93b419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4107360 0x7f93b419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 -- 172.21.15.23:0/422232319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 msgr2=0x7f93b41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:09.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 0x7f93b41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 -- 172.21.15.23:0/422232319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a40095f0 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b2ffd700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 0x7f93b41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b27fc700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f939c00e940 tx=0x7f939c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c009730 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b419da90 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f939c010430 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c018680 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.103+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b41a52f0 con 0x7f93b406b130 2024-03-16T20:35:09.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.105+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b410a530 con 0x7f93b406b130 2024-03-16T20:35:09.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.105+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f939c0108a0 con 0x7f93b406b130 2024-03-16T20:35:09.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.106+0000 7f93abfff700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 0x7f93a00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:09.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.106+0000 7f93b2ffd700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 0x7f93a00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:09.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.106+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f939c014070 con 0x7f93b406b130 2024-03-16T20:35:09.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.106+0000 7f93b2ffd700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 0x7f93a00a4010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f93a400b2e0 tx=0x7f93a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.111+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f939c053280 con 0x7f93b406b130 2024-03-16T20:35:09.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.321+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f93b4199900 con 0x7f93b406b130 2024-03-16T20:35:09.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.321+0000 7f93abfff700 1 -- 172.21.15.23:0/422232319 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f939c0967e0 con 0x7f93b406b130 2024-03-16T20:35:09.327 INFO:teuthology.orchestra.run.smithi023.stdout:347892351005 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 msgr2=0x7f93a00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 0x7f93a00a4010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f93a400b2e0 tx=0x7f93a4005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 msgr2=0x7f93b4198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f939c00e940 tx=0x7f939c00ec50 comp rx=0 tx=0).stop 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 shutdown_connections 2024-03-16T20:35:09.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f93a00a1b50 0x7f93a00a4010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b4107360 0x7f93b419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93b406a780 0x7f93b41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 --2- 172.21.15.23:0/422232319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93b406b130 0x7f93b4198ae0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 >> 172.21.15.23:0/422232319 conn(0x7f93b4076f30 msgr2=0x7f93b40ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 shutdown_connections 2024-03-16T20:35:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:09.323+0000 7f93b9313700 1 -- 172.21.15.23:0/422232319 wait complete. 2024-03-16T20:35:09.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:09 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/422232319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T20:35:09.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:09 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/422232319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T20:35:09.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:09 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/422232319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-16T20:35:10.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:10 smithi023 ceph-mon[31879]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:10.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:10 smithi031 ceph-mon[36281]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:10.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:10 smithi189 ceph-mon[36655]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:11.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351001 got 347892351005 for osd.11 2024-03-16T20:35:11.962 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:12.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:12 smithi023 ceph-mon[31879]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:12.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:12 smithi031 ceph-mon[36281]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:12.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:12 smithi189 ceph-mon[36655]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:12.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 -- 172.21.15.23:0/2071312962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 msgr2=0x7f2e3010ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 --2- 172.21.15.23:0/2071312962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e3010ee90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800b0a0 tx=0x7f2e2800b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 -- 172.21.15.23:0/2071312962 shutdown_connections 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 --2- 172.21.15.23:0/2071312962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e3010ee90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 --2- 172.21.15.23:0/2071312962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e301080e0 0x7f2e301084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 --2- 172.21.15.23:0/2071312962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e30102030 0x7f2e301024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 -- 172.21.15.23:0/2071312962 >> 172.21.15.23:0/2071312962 conn(0x7f2e30079110 msgr2=0x7f2e30079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 -- 172.21.15.23:0/2071312962 shutdown_connections 2024-03-16T20:35:12.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.862+0000 7f2e35421700 1 -- 172.21.15.23:0/2071312962 wait complete. 2024-03-16T20:35:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.863+0000 7f2e35421700 1 Processor -- start 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.863+0000 7f2e35421700 1 -- start start 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 0x7f2e301985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e301080e0 0x7f2e3019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3019d4c0 con 0x7f2e301029f0 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3019d630 con 0x7f2e30102030 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3019d7f0 con 0x7f2e301080e0 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49128/0 (socket says 172.21.15.23:49128) 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 -- 172.21.15.23:0/3412886000 learned_addr learned my addr 172.21.15.23:0/3412886000 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2effd700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 0x7f2e301985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2f7fe700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e301080e0 0x7f2e3019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 -- 172.21.15.23:0/3412886000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e301080e0 msgr2=0x7f2e3019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e301080e0 0x7f2e3019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 -- 172.21.15.23:0/3412886000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 msgr2=0x7f2e301985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 0x7f2e301985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 -- 172.21.15.23:0/3412886000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28009d00 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2effd700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 0x7f2e301985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e2e7fc700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e9a0 tx=0x7f2e2400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24009790 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e24010430 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.864+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3019dad0 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.865+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24018680 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.865+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300743e0 con 0x7f2e301029f0 2024-03-16T20:35:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.865+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3010a450 con 0x7f2e301029f0 2024-03-16T20:35:12.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.867+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2e24010910 con 0x7f2e301029f0 2024-03-16T20:35:12.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.867+0000 7f2e17fff700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 0x7f2e180a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.867+0000 7f2e2effd700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 0x7f2e180a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.867+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f2e24014070 con 0x7f2e301029f0 2024-03-16T20:35:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.868+0000 7f2e2effd700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 0x7f2e180a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e20011490 tx=0x7f2e20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:12.871+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2e24096c60 con 0x7f2e301029f0 2024-03-16T20:35:13.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.084+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2e30199940 con 0x7f2e301029f0 2024-03-16T20:35:13.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.084+0000 7f2e17fff700 1 -- 172.21.15.23:0/3412886000 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2e24026090 con 0x7f2e301029f0 2024-03-16T20:35:13.090 INFO:teuthology.orchestra.run.smithi023.stdout:141733920834 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 msgr2=0x7f2e180a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 0x7f2e180a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e20011490 tx=0x7f2e20009260 comp rx=0 tx=0).stop 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 msgr2=0x7f2e30198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e9a0 tx=0x7f2e2400ecb0 comp rx=0 tx=0).stop 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 shutdown_connections 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2e180a1bd0 0x7f2e180a4090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e301080e0 0x7f2e3019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e30102030 0x7f2e301985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:13.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 --2- 172.21.15.23:0/3412886000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e301029f0 0x7f2e30198b20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:13.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 >> 172.21.15.23:0/3412886000 conn(0x7f2e30079110 msgr2=0x7f2e30100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:13.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 shutdown_connections 2024-03-16T20:35:13.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:13.086+0000 7f2e35421700 1 -- 172.21.15.23:0/3412886000 wait complete. 2024-03-16T20:35:13.296 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:13 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3412886000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T20:35:13.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:13 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3412886000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T20:35:13.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:13 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3412886000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- 172.21.15.23:0/73059595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 msgr2=0x7fc6d81024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 --2- 172.21.15.23:0/73059595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d81024b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0099c0 tx=0x7fc6cc009cd0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- 172.21.15.23:0/73059595 shutdown_connections 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 --2- 172.21.15.23:0/73059595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 0x7fc6d810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 --2- 172.21.15.23:0/73059595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 0x7fc6d81083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 --2- 172.21.15.23:0/73059595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d81024b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- 172.21.15.23:0/73059595 >> 172.21.15.23:0/73059595 conn(0x7fc6d8079110 msgr2=0x7fc6d8079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- 172.21.15.23:0/73059595 shutdown_connections 2024-03-16T20:35:14.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- 172.21.15.23:0/73059595 wait complete. 2024-03-16T20:35:14.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 Processor -- start 2024-03-16T20:35:14.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.182+0000 7fc6de86c700 1 -- start start 2024-03-16T20:35:14.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 0x7fc6d8074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 0x7fc6d8074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81a68f0 con 0x7fc6d8102030 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81a6a60 con 0x7fc6d8108010 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6de86c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d81a6c20 con 0x7fc6d81029f0 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49150/0 (socket says 172.21.15.23:49150) 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 -- 172.21.15.23:0/565562202 learned_addr learned my addr 172.21.15.23:0/565562202 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d77fe700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 0x7fc6d8074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6dce09700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 0x7fc6d8074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 -- 172.21.15.23:0/565562202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 msgr2=0x7fc6d8074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 0x7fc6d8074820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 -- 172.21.15.23:0/565562202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 msgr2=0x7fc6d8074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 0x7fc6d8074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 -- 172.21.15.23:0/565562202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0095f0 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.183+0000 7fc6d7fff700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800e9a0 tx=0x7fc6c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.184+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8009790 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.184+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d81a6f00 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.184+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6c8010430 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.184+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8018680 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.184+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d81a7310 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.185+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d810a4f0 con 0x7fc6d8102030 2024-03-16T20:35:14.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.187+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc6c8010910 con 0x7fc6d8102030 2024-03-16T20:35:14.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.187+0000 7fc6d57fa700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 0x7fc6c00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.188+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fc6c8014070 con 0x7fc6d8102030 2024-03-16T20:35:14.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.188+0000 7fc6d77fe700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 0x7fc6c00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.188+0000 7fc6d77fe700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 0x7fc6c00a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc000c00 tx=0x7fc6cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.192+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc6c8053280 con 0x7fc6d8102030 2024-03-16T20:35:14.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:14 smithi023 ceph-mon[31879]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:14.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.399+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fc6d81120b0 con 0x7fc6d8102030 2024-03-16T20:35:14.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.399+0000 7fc6d57fa700 1 -- 172.21.15.23:0/565562202 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc6c8096760 con 0x7fc6d8102030 2024-03-16T20:35:14.405 INFO:teuthology.orchestra.run.smithi023.stdout:313532612643 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.402+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 msgr2=0x7fc6c00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 0x7fc6c00a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc000c00 tx=0x7fc6cc005fb0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 msgr2=0x7fc6d80742e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800e9a0 tx=0x7fc6c800ecb0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 shutdown_connections 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc6c00a1b50 0x7fc6c00a4010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d8102030 0x7fc6d80742e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6d8108010 0x7fc6d8074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 --2- 172.21.15.23:0/565562202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d81029f0 0x7fc6d8074820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 >> 172.21.15.23:0/565562202 conn(0x7fc6d8079110 msgr2=0x7fc6d8100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:14.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 shutdown_connections 2024-03-16T20:35:14.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.403+0000 7fc6de86c700 1 -- 172.21.15.23:0/565562202 wait complete. 2024-03-16T20:35:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:14 smithi031 ceph-mon[36281]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:14.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:14 smithi189 ceph-mon[36655]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 -- 172.21.15.23:0/2677618165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1458107360 msgr2=0x7f145810eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 --2- 172.21.15.23:0/2677618165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1458107360 0x7f145810eee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f145000b0a0 tx=0x7f145000b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 -- 172.21.15.23:0/2677618165 shutdown_connections 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 --2- 172.21.15.23:0/2677618165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1458107360 0x7f145810eee0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 --2- 172.21.15.23:0/2677618165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145806a9a0 0x7f145806ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 --2- 172.21.15.23:0/2677618165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806b350 0x7f1458106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 -- 172.21.15.23:0/2677618165 >> 172.21.15.23:0/2677618165 conn(0x7f1458076f20 msgr2=0x7f1458077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 -- 172.21.15.23:0/2677618165 shutdown_connections 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.615+0000 7f145f311700 1 -- 172.21.15.23:0/2677618165 wait complete. 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 Processor -- start 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 -- start start 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806a9a0 0x7f1458100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458107360 0x7f1458101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581a6a50 con 0x7f145806b350 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581a6bc0 con 0x7f1458107360 2024-03-16T20:35:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145f311700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581a6d80 con 0x7f145806a9a0 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145c8ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145d8ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458107360 0x7f1458101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145d0ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806a9a0 0x7f1458100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145c8ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49168/0 (socket says 172.21.15.23:49168) 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.616+0000 7f145c8ac700 1 -- 172.21.15.23:0/28524333 learned_addr learned my addr 172.21.15.23:0/28524333 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 -- 172.21.15.23:0/28524333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806a9a0 msgr2=0x7f1458100fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806a9a0 0x7f1458100fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 -- 172.21.15.23:0/28524333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458107360 msgr2=0x7f1458101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458107360 0x7f1458101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 -- 172.21.15.23:0/28524333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1450009d00 con 0x7f145806b350 2024-03-16T20:35:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145c8ac700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f144800e940 tx=0x7f144800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448009730 con 0x7f145806b350 2024-03-16T20:35:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1448010430 con 0x7f145806b350 2024-03-16T20:35:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14581a7060 con 0x7f145806b350 2024-03-16T20:35:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448018660 con 0x7f145806b350 2024-03-16T20:35:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.617+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581a75b0 con 0x7f145806b350 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.618+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145810a550 con 0x7f145806b350 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.619+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f14480108a0 con 0x7f145806b350 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.620+0000 7f144e7fc700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 0x7f14440a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.620+0000 7f145d0ad700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 0x7f14440a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.620+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f1448014070 con 0x7f145806b350 2024-03-16T20:35:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.621+0000 7f145d0ad700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 0x7f14440a4090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1454009c60 tx=0x7f1454009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.626+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1448053280 con 0x7f145806b350 2024-03-16T20:35:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 -- 172.21.15.23:0/2585308431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988102bd0 msgr2=0x7f0988106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 --2- 172.21.15.23:0/2585308431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988102bd0 0x7f0988106c20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09740098f0 tx=0x7f0974009c00 comp rx=0 tx=0).stop 2024-03-16T20:35:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 -- 172.21.15.23:0/2585308431 shutdown_connections 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 --2- 172.21.15.23:0/2585308431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988107300 0x7f098810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 --2- 172.21.15.23:0/2585308431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 0x7f0988102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 --2- 172.21.15.23:0/2585308431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988102bd0 0x7f0988106c20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 -- 172.21.15.23:0/2585308431 >> 172.21.15.23:0/2585308431 conn(0x7f09880fda90 msgr2=0x7f09880ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 -- 172.21.15.23:0/2585308431 shutdown_connections 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.818+0000 7f098ecf8700 1 -- 172.21.15.23:0/2585308431 wait complete. 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 Processor -- start 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 -- start start 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 0x7f09880742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 0x7f0988074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881a6990 con 0x7f0988102bd0 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881a6b00 con 0x7f0988107300 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ecf8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881a6cc0 con 0x7f0988102220 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098ca94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 0x7f09880742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49182/0 (socket says 172.21.15.23:49182) 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098d295700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 0x7f0988074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 -- 172.21.15.23:0/3629362426 learned_addr learned my addr 172.21.15.23:0/3629362426 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 -- 172.21.15.23:0/3629362426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 msgr2=0x7f09880742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 0x7f09880742e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 -- 172.21.15.23:0/3629362426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 msgr2=0x7f0988074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 0x7f0988074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 -- 172.21.15.23:0/3629362426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0980009d00 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f098d295700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 0x7f0988074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.819+0000 7f0987fff700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f097400bf50 tx=0x7f097400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.820+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097401c070 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.820+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f097400b560 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.820+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0974009520 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.820+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974003c40 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.820+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09881a71c0 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.821+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098810a620 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.822+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f097400b6d0 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.823+0000 7f0985ffb700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 0x7f09780a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.823+0000 7f098ca94700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 0x7f09780a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.823+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f09740ce800 con 0x7f0988102bd0 2024-03-16T20:35:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.823+0000 7f098ca94700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 0x7f09780a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f097c009c60 tx=0x7f097c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.830+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0974097600 con 0x7f0988102bd0 2024-03-16T20:35:14.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.837+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f1458068ca0 con 0x7f145806b350 2024-03-16T20:35:14.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.838+0000 7f144e7fc700 1 -- 172.21.15.23:0/28524333 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f14480966d0 con 0x7f145806b350 2024-03-16T20:35:14.844 INFO:teuthology.orchestra.run.smithi023.stdout:257698037806 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 msgr2=0x7f14440a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 0x7f14440a4090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1454009c60 tx=0x7f1454009260 comp rx=0 tx=0).stop 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 msgr2=0x7f1458101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f144800e940 tx=0x7f144800ec50 comp rx=0 tx=0).stop 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 shutdown_connections 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f14440a1bd0 0x7f14440a4090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458107360 0x7f1458101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f145806a9a0 0x7f1458100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 --2- 172.21.15.23:0/28524333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f145806b350 0x7f1458101520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:14.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 >> 172.21.15.23:0/28524333 conn(0x7f1458076f20 msgr2=0x7f14580ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:14.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 shutdown_connections 2024-03-16T20:35:14.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:14.840+0000 7f145f311700 1 -- 172.21.15.23:0/28524333 wait complete. 2024-03-16T20:35:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.039+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f09881120d0 con 0x7f0988102bd0 2024-03-16T20:35:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.040+0000 7f0985ffb700 1 -- 172.21.15.23:0/3629362426 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f097409acc0 con 0x7f0988102bd0 2024-03-16T20:35:15.046 INFO:teuthology.orchestra.run.smithi023.stdout:219043332146 2024-03-16T20:35:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 msgr2=0x7f09780a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 0x7f09780a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f097c009c60 tx=0x7f097c009260 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 msgr2=0x7f0988074820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f097400bf50 tx=0x7f097400fa70 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 shutdown_connections 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f09780a1b50 0x7f09780a4010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0988107300 0x7f0988074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0988102220 0x7f09880742e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 --2- 172.21.15.23:0/3629362426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0988102bd0 0x7f0988074820 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 >> 172.21.15.23:0/3629362426 conn(0x7f09880fda90 msgr2=0x7f09880ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.042+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 shutdown_connections 2024-03-16T20:35:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:15.043+0000 7f098ecf8700 1 -- 172.21.15.23:0/3629362426 wait complete. 2024-03-16T20:35:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/565562202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T20:35:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/28524333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T20:35:15.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:15 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3629362426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T20:35:15.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/565562202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T20:35:15.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/28524333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T20:35:15.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:15 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3629362426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T20:35:15.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/565562202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-16T20:35:15.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/28524333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-16T20:35:15.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:15 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3629362426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 -- 172.21.15.23:0/2037463545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 msgr2=0x7ff14c101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 --2- 172.21.15.23:0/2037463545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c101640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff1400099c0 tx=0x7ff140009cd0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 -- 172.21.15.23:0/2037463545 shutdown_connections 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 --2- 172.21.15.23:0/2037463545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 0x7ff14c10dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 --2- 172.21.15.23:0/2037463545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 0x7ff14c1073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 --2- 172.21.15.23:0/2037463545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c101640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 -- 172.21.15.23:0/2037463545 >> 172.21.15.23:0/2037463545 conn(0x7ff14c077da0 msgr2=0x7ff14c0781b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 -- 172.21.15.23:0/2037463545 shutdown_connections 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.233+0000 7ff1532dc700 1 -- 172.21.15.23:0/2037463545 wait complete. 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 Processor -- start 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 -- start start 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 0x7ff14c10ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 0x7ff14c10f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1a68c0 con 0x7ff14c1011c0 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1a6a30 con 0x7ff14c107010 2024-03-16T20:35:16.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff1532dc700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c1a6bf0 con 0x7ff14c101b80 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49196/0 (socket says 172.21.15.23:49196) 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 -- 172.21.15.23:0/195683717 learned_addr learned my addr 172.21.15.23:0/195683717 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff150877700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 0x7ff14c10ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151879700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 0x7ff14c10f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 -- 172.21.15.23:0/195683717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 msgr2=0x7ff14c10ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 0x7ff14c10ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 -- 172.21.15.23:0/195683717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 msgr2=0x7ff14c10f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 0x7ff14c10f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 -- 172.21.15.23:0/195683717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c009520 con 0x7ff14c1011c0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.234+0000 7ff151078700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff14400ef10 tx=0x7ff14400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.235+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400ca70 con 0x7ff14c1011c0 2024-03-16T20:35:16.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.235+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1400095f0 con 0x7ff14c1011c0 2024-03-16T20:35:16.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.235+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff14400f040 con 0x7ff14c1011c0 2024-03-16T20:35:16.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.235+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1440146e0 con 0x7ff14c1011c0 2024-03-16T20:35:16.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.235+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1a71b0 con 0x7ff14c1011c0 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.236+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c1093a0 con 0x7ff14c1011c0 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.237+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff144007500 con 0x7ff14c1011c0 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.237+0000 7ff14a7fc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 0x7ff1340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.237+0000 7ff150877700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 0x7ff1340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.237+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff1440cdc00 con 0x7ff14c1011c0 2024-03-16T20:35:16.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.238+0000 7ff150877700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 0x7ff1340a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff140000c00 tx=0x7ff140005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:16.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.242+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff144096970 con 0x7ff14c1011c0 2024-03-16T20:35:16.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:16 smithi023 ceph-mon[31879]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:16.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:16 smithi031 ceph-mon[36281]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.455+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff14c10fd00 con 0x7ff14c1011c0 2024-03-16T20:35:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.455+0000 7ff14a7fc700 1 -- 172.21.15.23:0/195683717 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff14409a030 con 0x7ff14c1011c0 2024-03-16T20:35:16.461 INFO:teuthology.orchestra.run.smithi023.stdout:163208757309 2024-03-16T20:35:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 msgr2=0x7ff1340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 0x7ff1340a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff140000c00 tx=0x7ff140005fb0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 msgr2=0x7ff14c10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff14400ef10 tx=0x7ff14400c3a0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 shutdown_connections 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff1340a1b50 0x7ff1340a4010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff14c1011c0 0x7ff14c10e770 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff14c107010 0x7ff14c10f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 --2- 172.21.15.23:0/195683717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff14c101b80 0x7ff14c10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 >> 172.21.15.23:0/195683717 conn(0x7ff14c077da0 msgr2=0x7ff14c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 shutdown_connections 2024-03-16T20:35:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:16.457+0000 7ff1532dc700 1 -- 172.21.15.23:0/195683717 wait complete. 2024-03-16T20:35:16.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:16 smithi189 ceph-mon[36655]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:16.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920834 for osd.4 2024-03-16T20:35:16.693 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:17.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332141 got 219043332146 for osd.7 2024-03-16T20:35:17.363 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:17.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:17 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/195683717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T20:35:17.448 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:17 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/195683717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T20:35:17.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:17 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/195683717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T20:35:17.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612639 got 313532612643 for osd.10 2024-03-16T20:35:17.825 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:18.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:18 smithi031 ceph-mon[36281]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:18.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:18 smithi023 ceph-mon[31879]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:18 smithi189 ceph-mon[36655]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:18.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757300 got 163208757309 for osd.5 2024-03-16T20:35:18.807 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:19.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037801 got 257698037806 for osd.8 2024-03-16T20:35:19.261 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 -- 172.21.15.23:0/1985899516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087806b130 msgr2=0x7f0878106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 --2- 172.21.15.23:0/1985899516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087806b130 0x7f0878106be0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f086c009990 tx=0x7f086c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 -- 172.21.15.23:0/1985899516 shutdown_connections 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 --2- 172.21.15.23:0/1985899516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08781072c0 0x7f087810ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 --2- 172.21.15.23:0/1985899516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806a780 0x7f087806ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 --2- 172.21.15.23:0/1985899516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087806b130 0x7f0878106be0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 -- 172.21.15.23:0/1985899516 >> 172.21.15.23:0/1985899516 conn(0x7f0878076f30 msgr2=0x7f0878077340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 -- 172.21.15.23:0/1985899516 shutdown_connections 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.475+0000 7f087dd28700 1 -- 172.21.15.23:0/1985899516 wait complete. 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 Processor -- start 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 -- start start 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 0x7f0878198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819d3e0 con 0x7f08781072c0 2024-03-16T20:35:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819d550 con 0x7f087806b130 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.476+0000 7f087dd28700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819d710 con 0x7f087806a780 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f08777fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 0x7f0878198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49212/0 (socket says 172.21.15.23:49212) 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:50052/0 (socket says 172.21.15.23:50052) 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 -- 172.21.15.23:0/2315174908 learned_addr learned my addr 172.21.15.23:0/2315174908 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 -- 172.21.15.23:0/2315174908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 msgr2=0x7f0878198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 0x7f0878198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 -- 172.21.15.23:0/2315174908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 msgr2=0x7f0878198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 -- 172.21.15.23:0/2315174908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0095f0 con 0x7f08781072c0 2024-03-16T20:35:19.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0876ffd700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f08777fe700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 0x7f0878198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0877fff700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f087000ef30 tx=0x7f087000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870018070 con 0x7f08781072c0 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f087000f040 con 0x7f08781072c0 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087819d990 con 0x7f08781072c0 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000cd50 con 0x7f08781072c0 2024-03-16T20:35:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.477+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08781a5250 con 0x7f08781072c0 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.479+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087810a5e0 con 0x7f08781072c0 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.480+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0870007500 con 0x7f08781072c0 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.480+0000 7f0874ff9700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 0x7f08600a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.481+0000 7f08777fe700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 0x7f08600a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.481+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f08700cd360 con 0x7f08781072c0 2024-03-16T20:35:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.481+0000 7f08777fe700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 0x7f08600a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0868011490 tx=0x7f0868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:19.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.485+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0870096160 con 0x7f08781072c0 2024-03-16T20:35:19.789 INFO:teuthology.orchestra.run.smithi023.stdout:197568495673 2024-03-16T20:35:19.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.783+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0878199860 con 0x7f08781072c0 2024-03-16T20:35:19.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.783+0000 7f0874ff9700 1 -- 172.21.15.23:0/2315174908 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0870099820 con 0x7f08781072c0 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 msgr2=0x7f08600a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 0x7f08600a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0868011490 tx=0x7f0868009260 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 msgr2=0x7f087819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f087000ef30 tx=0x7f087000c3a0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 shutdown_connections 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f08600a1b50 0x7f08600a4010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08781072c0 0x7f087819cdd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f087806a780 0x7f0878198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 --2- 172.21.15.23:0/2315174908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f087806b130 0x7f0878198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 >> 172.21.15.23:0/2315174908 conn(0x7f0878076f30 msgr2=0x7f08780fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 shutdown_connections 2024-03-16T20:35:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:19.786+0000 7f087dd28700 1 -- 172.21.15.23:0/2315174908 wait complete. 2024-03-16T20:35:20.456 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495667 got 197568495673 for osd.6 2024-03-16T20:35:20.457 DEBUG:teuthology.parallel:result is None 2024-03-16T20:35:20.457 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T20:35:20.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T20:35:20.711 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:20.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:20 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:20 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2315174908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T20:35:20.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:20 smithi031 ceph-mon[36281]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:20.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:20 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:20 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2315174908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T20:35:20.970 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:20 smithi023 ceph-mon[31879]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:20 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:20 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2315174908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T20:35:20.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:20 smithi189 ceph-mon[36655]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:21.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:35:21.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:35:21.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:22.001 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:35:22.001 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:35:22.001 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:22.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:35:22.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:35:22.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:35:22.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.355+0000 7f8d68540700 1 -- 172.21.15.23:0/1238680534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60108100 msgr2=0x7f8d601084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.355+0000 7f8d68540700 1 --2- 172.21.15.23:0/1238680534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60108100 0x7f8d601084e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009990 tx=0x7f8d5c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 -- 172.21.15.23:0/1238680534 shutdown_connections 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- 172.21.15.23:0/1238680534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60108100 0x7f8d601084e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- 172.21.15.23:0/1238680534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60102a50 0x7f8d6010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- 172.21.15.23:0/1238680534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102090 0x7f8d60102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 -- 172.21.15.23:0/1238680534 >> 172.21.15.23:0/1238680534 conn(0x7f8d600790b0 msgr2=0x7f8d600794c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 -- 172.21.15.23:0/1238680534 shutdown_connections 2024-03-16T20:35:22.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 -- 172.21.15.23:0/1238680534 wait complete. 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 Processor -- start 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 -- start start 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102a50 0x7f8d60074820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.356+0000 7f8d68540700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60108100 0x7f8d60074d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d68540700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a69a0 con 0x7f8d60102090 2024-03-16T20:35:22.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d68540700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a6b10 con 0x7f8d60102a50 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d68540700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a6cd0 con 0x7f8d60108100 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47902/0 (socket says 172.21.15.23:47902) 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 -- 172.21.15.23:0/1027732352 learned_addr learned my addr 172.21.15.23:0/1027732352 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d65adb700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102a50 0x7f8d60074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d66add700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60108100 0x7f8d60074d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 -- 172.21.15.23:0/1027732352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60108100 msgr2=0x7f8d60074d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60108100 0x7f8d60074d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 -- 172.21.15.23:0/1027732352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102a50 msgr2=0x7f8d60074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102a50 0x7f8d60074820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 -- 172.21.15.23:0/1027732352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0095f0 con 0x7f8d60102090 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d662dc700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c01d070 con 0x7f8d60102090 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c00b610 con 0x7f8d60102090 2024-03-16T20:35:22.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d601a6f50 con 0x7f8d60102090 2024-03-16T20:35:22.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0173f0 con 0x7f8d60102090 2024-03-16T20:35:22.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.357+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601a7360 con 0x7f8d60102090 2024-03-16T20:35:22.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.358+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6010a520 con 0x7f8d60102090 2024-03-16T20:35:22.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.359+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8d5c022670 con 0x7f8d60102090 2024-03-16T20:35:22.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.360+0000 7f8d577fe700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 0x7f8d4c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:22.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.360+0000 7f8d65adb700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 0x7f8d4c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:22.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.360+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f8d5c0ce7d0 con 0x7f8d60102090 2024-03-16T20:35:22.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.360+0000 7f8d65adb700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 0x7f8d4c0a4000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d60106ff0 tx=0x7f8d50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:22.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.364+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8d5c097730 con 0x7f8d60102090 2024-03-16T20:35:22.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.563+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d601120a0 con 0x7f8d4c0a1b40 2024-03-16T20:35:22.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.572+0000 7f8d577fe700 1 -- 172.21.15.23:0/1027732352 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340846 (secure 0 0 0) 0x7f8d601120a0 con 0x7f8d4c0a1b40 2024-03-16T20:35:22.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 msgr2=0x7f8d4c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 0x7f8d4c0a4000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d60106ff0 tx=0x7f8d50009260 comp rx=0 tx=0).stop 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 msgr2=0x7f8d600742e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c000c00 tx=0x7f8d5c004730 comp rx=0 tx=0).stop 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 shutdown_connections 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f8d4c0a1b40 0x7f8d4c0a4000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102090 0x7f8d600742e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d60108100 0x7f8d60074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 --2- 172.21.15.23:0/1027732352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d60102a50 0x7f8d60074820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 >> 172.21.15.23:0/1027732352 conn(0x7f8d600790b0 msgr2=0x7f8d60100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 shutdown_connections 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:22.577+0000 7f8d68540700 1 -- 172.21.15.23:0/1027732352 wait complete. 2024-03-16T20:35:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-16T20:35:22.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:22 smithi023 ceph-mon[31879]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T20:35:22.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:22 smithi023 ceph-mon[31879]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-16T20:35:23.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:22 smithi031 ceph-mon[36281]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T20:35:23.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:22 smithi031 ceph-mon[36281]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-16T20:35:23.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:22 smithi189 ceph-mon[36655]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-16T20:35:23.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:22 smithi189 ceph-mon[36655]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-16T20:35:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-16T20:35:21.964469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658886,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":878,"num_read_kb":563,"num_write":427,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":642,"ondisk_log_size":642,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378276,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563548,"statfs":{"total":1151940427776,"available":1151553073152,"internally_reserved":0,"allocated":12738560,"data_stored":8030459,"data_compressed":357126,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":76,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415995+0000","last_change":"2024-03-16T20:33:49.858111+0000","last_active":"2024-03-16T20:34:51.415995+0000","last_peered":"2024-03-16T20:34:51.415995+0000","last_clean":"2024-03-16T20:34:51.415995+0000","last_became_active":"2024-03-16T20:33:49.857656+0000","last_became_peered":"2024-03-16T20:33:49.857656+0000","last_unstale":"2024-03-16T20:34:51.415995+0000","last_undegraded":"2024-03-16T20:34:51.415995+0000","last_fullsized":"2024-03-16T20:34:51.415995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:16:47.779595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414266+0000","last_change":"2024-03-16T20:33:49.854850+0000","last_active":"2024-03-16T20:34:51.414266+0000","last_peered":"2024-03-16T20:34:51.414266+0000","last_clean":"2024-03-16T20:34:51.414266+0000","last_became_active":"2024-03-16T20:33:49.854399+0000","last_became_peered":"2024-03-16T20:33:49.854399+0000","last_unstale":"2024-03-16T20:34:51.414266+0000","last_undegraded":"2024-03-16T20:34:51.414266+0000","last_fullsized":"2024-03-16T20:34:51.414266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:55.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414185+0000","last_change":"2024-03-16T20:33:49.858831+0000","last_active":"2024-03-16T20:34:51.414185+0000","last_peered":"2024-03-16T20:34:51.414185+0000","last_clean":"2024-03-16T20:34:51.414185+0000","last_became_active":"2024-03-16T20:33:49.858507+0000","last_became_peered":"2024-03-16T20:33:49.858507+0000","last_unstale":"2024-03-16T20:34:51.414185+0000","last_undegraded":"2024-03-16T20:34:51.414185+0000","last_fullsized":"2024-03-16T20:34:51.414185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:35.447285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979087+0000","last_change":"2024-03-16T20:33:49.852269+0000","last_active":"2024-03-16T20:35:19.979087+0000","last_peered":"2024-03-16T20:35:19.979087+0000","last_clean":"2024-03-16T20:35:19.979087+0000","last_became_active":"2024-03-16T20:33:49.851904+0000","last_became_peered":"2024-03-16T20:33:49.851904+0000","last_unstale":"2024-03-16T20:35:19.979087+0000","last_undegraded":"2024-03-16T20:35:19.979087+0000","last_fullsized":"2024-03-16T20:35:19.979087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:38.520111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.961191+0000","last_change":"2024-03-16T20:33:49.852371+0000","last_active":"2024-03-16T20:35:19.961191+0000","last_peered":"2024-03-16T20:35:19.961191+0000","last_clean":"2024-03-16T20:35:19.961191+0000","last_became_active":"2024-03-16T20:33:49.852034+0000","last_became_peered":"2024-03-16T20:33:49.852034+0000","last_unstale":"2024-03-16T20:35:19.961191+0000","last_undegraded":"2024-03-16T20:35:19.961191+0000","last_fullsized":"2024-03-16T20:35:19.961191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:10.229852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.971913+0000","last_change":"2024-03-16T20:33:49.855209+0000","last_active":"2024-03-16T20:35:19.971913+0000","last_peered":"2024-03-16T20:35:19.971913+0000","last_clean":"2024-03-16T20:35:19.971913+0000","last_became_active":"2024-03-16T20:33:49.854502+0000","last_became_peered":"2024-03-16T20:33:49.854502+0000","last_unstale":"2024-03-16T20:35:19.971913+0000","last_undegraded":"2024-03-16T20:35:19.971913+0000","last_fullsized":"2024-03-16T20:35:19.971913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:47.792663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417926+0000","last_change":"2024-03-16T20:33:49.852668+0000","last_active":"2024-03-16T20:34:51.417926+0000","last_peered":"2024-03-16T20:34:51.417926+0000","last_clean":"2024-03-16T20:34:51.417926+0000","last_became_active":"2024-03-16T20:33:49.852263+0000","last_became_peered":"2024-03-16T20:33:49.852263+0000","last_unstale":"2024-03-16T20:34:51.417926+0000","last_undegraded":"2024-03-16T20:34:51.417926+0000","last_fullsized":"2024-03-16T20:34:51.417926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:49.815894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.981712+0000","last_change":"2024-03-16T20:33:49.854962+0000","last_active":"2024-03-16T20:35:19.981712+0000","last_peered":"2024-03-16T20:35:19.981712+0000","last_clean":"2024-03-16T20:35:19.981712+0000","last_became_active":"2024-03-16T20:33:49.853963+0000","last_became_peered":"2024-03-16T20:33:49.853963+0000","last_unstale":"2024-03-16T20:35:19.981712+0000","last_undegraded":"2024-03-16T20:35:19.981712+0000","last_fullsized":"2024-03-16T20:35:19.981712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:37:19.213607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.992384+0000","last_change":"2024-03-16T20:33:49.857741+0000","last_active":"2024-03-16T20:35:19.992384+0000","last_peered":"2024-03-16T20:35:19.992384+0000","last_clean":"2024-03-16T20:35:19.992384+0000","last_became_active":"2024-03-16T20:33:49.857354+0000","last_became_peered":"2024-03-16T20:33:49.857354+0000","last_unstale":"2024-03-16T20:35:19.992384+0000","last_undegraded":"2024-03-16T20:35:19.992384+0000","last_fullsized":"2024-03-16T20:35:19.992384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:22.085460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.982596+0000","last_change":"2024-03-16T20:33:49.864008+0000","last_active":"2024-03-16T20:35:19.982596+0000","last_peered":"2024-03-16T20:35:19.982596+0000","last_clean":"2024-03-16T20:35:19.982596+0000","last_became_active":"2024-03-16T20:33:49.862779+0000","last_became_peered":"2024-03-16T20:33:49.862779+0000","last_unstale":"2024-03-16T20:35:19.982596+0000","last_undegraded":"2024-03-16T20:35:19.982596+0000","last_fullsized":"2024-03-16T20:35:19.982596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:34:23.249973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419001+0000","last_change":"2024-03-16T20:33:49.858951+0000","last_active":"2024-03-16T20:34:51.419001+0000","last_peered":"2024-03-16T20:34:51.419001+0000","last_clean":"2024-03-16T20:34:51.419001+0000","last_became_active":"2024-03-16T20:33:49.858575+0000","last_became_peered":"2024-03-16T20:33:49.858575+0000","last_unstale":"2024-03-16T20:34:51.419001+0000","last_undegraded":"2024-03-16T20:34:51.419001+0000","last_fullsized":"2024-03-16T20:34:51.419001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:07:01.030826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.960235+0000","last_change":"2024-03-16T20:33:49.854399+0000","last_active":"2024-03-16T20:35:19.960235+0000","last_peered":"2024-03-16T20:35:19.960235+0000","last_clean":"2024-03-16T20:35:19.960235+0000","last_became_active":"2024-03-16T20:33:49.853937+0000","last_became_peered":"2024-03-16T20:33:49.853937+0000","last_unstale":"2024-03-16T20:35:19.960235+0000","last_undegraded":"2024-03-16T20:35:19.960235+0000","last_fullsized":"2024-03-16T20:35:19.960235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:48:00.207213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983196+0000","last_change":"2024-03-16T20:33:49.853101+0000","last_active":"2024-03-16T20:35:19.983196+0000","last_peered":"2024-03-16T20:35:19.983196+0000","last_clean":"2024-03-16T20:35:19.983196+0000","last_became_active":"2024-03-16T20:33:49.852667+0000","last_became_peered":"2024-03-16T20:33:49.852667+0000","last_unstale":"2024-03-16T20:35:19.983196+0000","last_undegraded":"2024-03-16T20:35:19.983196+0000","last_fullsized":"2024-03-16T20:35:19.983196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:17:43.047348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413943+0000","last_change":"2024-03-16T20:33:49.852907+0000","last_active":"2024-03-16T20:34:51.413943+0000","last_peered":"2024-03-16T20:34:51.413943+0000","last_clean":"2024-03-16T20:34:51.413943+0000","last_became_active":"2024-03-16T20:33:49.852552+0000","last_became_peered":"2024-03-16T20:33:49.852552+0000","last_unstale":"2024-03-16T20:34:51.413943+0000","last_undegraded":"2024-03-16T20:34:51.413943+0000","last_fullsized":"2024-03-16T20:34:51.413943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:46:34.019736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414017+0000","last_change":"2024-03-16T20:33:49.863477+0000","last_active":"2024-03-16T20:34:51.414017+0000","last_peered":"2024-03-16T20:34:51.414017+0000","last_clean":"2024-03-16T20:34:51.414017+0000","last_became_active":"2024-03-16T20:33:49.863163+0000","last_became_peered":"2024-03-16T20:33:49.863163+0000","last_unstale":"2024-03-16T20:34:51.414017+0000","last_undegraded":"2024-03-16T20:34:51.414017+0000","last_fullsized":"2024-03-16T20:34:51.414017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:46:55.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994338+0000","last_change":"2024-03-16T20:33:49.863312+0000","last_active":"2024-03-16T20:35:19.994338+0000","last_peered":"2024-03-16T20:35:19.994338+0000","last_clean":"2024-03-16T20:35:19.994338+0000","last_became_active":"2024-03-16T20:33:49.862918+0000","last_became_peered":"2024-03-16T20:33:49.862918+0000","last_unstale":"2024-03-16T20:35:19.994338+0000","last_undegraded":"2024-03-16T20:35:19.994338+0000","last_fullsized":"2024-03-16T20:35:19.994338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:33.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415760+0000","last_change":"2024-03-16T20:33:49.858863+0000","last_active":"2024-03-16T20:34:51.415760+0000","last_peered":"2024-03-16T20:34:51.415760+0000","last_clean":"2024-03-16T20:34:51.415760+0000","last_became_active":"2024-03-16T20:33:49.858543+0000","last_became_peered":"2024-03-16T20:33:49.858543+0000","last_unstale":"2024-03-16T20:34:51.415760+0000","last_undegraded":"2024-03-16T20:34:51.415760+0000","last_fullsized":"2024-03-16T20:34:51.415760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:04.584473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976378+0000","last_change":"2024-03-16T20:33:49.857863+0000","last_active":"2024-03-16T20:35:19.976378+0000","last_peered":"2024-03-16T20:35:19.976378+0000","last_clean":"2024-03-16T20:35:19.976378+0000","last_became_active":"2024-03-16T20:33:49.857503+0000","last_became_peered":"2024-03-16T20:33:49.857503+0000","last_unstale":"2024-03-16T20:35:19.976378+0000","last_undegraded":"2024-03-16T20:35:19.976378+0000","last_fullsized":"2024-03-16T20:35:19.976378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:11.206926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.977578+0000","last_change":"2024-03-16T20:33:49.849940+0000","last_active":"2024-03-16T20:35:19.977578+0000","last_peered":"2024-03-16T20:35:19.977578+0000","last_clean":"2024-03-16T20:35:19.977578+0000","last_became_active":"2024-03-16T20:33:49.849632+0000","last_became_peered":"2024-03-16T20:33:49.849632+0000","last_unstale":"2024-03-16T20:35:19.977578+0000","last_undegraded":"2024-03-16T20:35:19.977578+0000","last_fullsized":"2024-03-16T20:35:19.977578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:25.921270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418880+0000","last_change":"2024-03-16T20:33:53.882456+0000","last_active":"2024-03-16T20:34:51.418880+0000","last_peered":"2024-03-16T20:34:51.418880+0000","last_clean":"2024-03-16T20:34:51.418880+0000","last_became_active":"2024-03-16T20:33:53.882040+0000","last_became_peered":"2024-03-16T20:33:53.882040+0000","last_unstale":"2024-03-16T20:34:51.418880+0000","last_undegraded":"2024-03-16T20:34:51.418880+0000","last_fullsized":"2024-03-16T20:34:51.418880+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:59.392600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":112,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:20.037290+0000","last_change":"2024-03-16T20:33:48.846129+0000","last_active":"2024-03-16T20:35:20.037290+0000","last_peered":"2024-03-16T20:35:20.037290+0000","last_clean":"2024-03-16T20:35:20.037290+0000","last_became_active":"2024-03-16T20:33:48.845783+0000","last_became_peered":"2024-03-16T20:33:48.845783+0000","last_unstale":"2024-03-16T20:35:20.037290+0000","last_undegraded":"2024-03-16T20:35:20.037290+0000","last_fullsized":"2024-03-16T20:35:20.037290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:46:03.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.973691+0000","last_change":"2024-03-16T20:33:49.851397+0000","last_active":"2024-03-16T20:35:19.973691+0000","last_peered":"2024-03-16T20:35:19.973691+0000","last_clean":"2024-03-16T20:35:19.973691+0000","last_became_active":"2024-03-16T20:33:49.851097+0000","last_became_peered":"2024-03-16T20:33:49.851097+0000","last_unstale":"2024-03-16T20:35:19.973691+0000","last_undegraded":"2024-03-16T20:35:19.973691+0000","last_fullsized":"2024-03-16T20:35:19.973691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:57:55.165017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414413+0000","last_change":"2024-03-16T20:33:53.893593+0000","last_active":"2024-03-16T20:34:51.414413+0000","last_peered":"2024-03-16T20:34:51.414413+0000","last_clean":"2024-03-16T20:34:51.414413+0000","last_became_active":"2024-03-16T20:33:53.893101+0000","last_became_peered":"2024-03-16T20:33:53.893101+0000","last_unstale":"2024-03-16T20:34:51.414413+0000","last_undegraded":"2024-03-16T20:34:51.414413+0000","last_fullsized":"2024-03-16T20:34:51.414413+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:22:59.472049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418047+0000","last_change":"2024-03-16T20:33:48.852786+0000","last_active":"2024-03-16T20:34:51.418047+0000","last_peered":"2024-03-16T20:34:51.418047+0000","last_clean":"2024-03-16T20:34:51.418047+0000","last_became_active":"2024-03-16T20:33:48.852477+0000","last_became_peered":"2024-03-16T20:33:48.852477+0000","last_unstale":"2024-03-16T20:34:51.418047+0000","last_undegraded":"2024-03-16T20:34:51.418047+0000","last_fullsized":"2024-03-16T20:34:51.418047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:49.191312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414490+0000","last_change":"2024-03-16T20:33:49.859123+0000","last_active":"2024-03-16T20:34:51.414490+0000","last_peered":"2024-03-16T20:34:51.414490+0000","last_clean":"2024-03-16T20:34:51.414490+0000","last_became_active":"2024-03-16T20:33:49.858746+0000","last_became_peered":"2024-03-16T20:33:49.858746+0000","last_unstale":"2024-03-16T20:34:51.414490+0000","last_undegraded":"2024-03-16T20:34:51.414490+0000","last_fullsized":"2024-03-16T20:34:51.414490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:35.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.910052+0000","last_change":"2024-03-16T20:34:51.910052+0000","last_active":"2024-03-16T20:34:51.910052+0000","last_peered":"2024-03-16T20:34:51.910052+0000","last_clean":"2024-03-16T20:34:51.910052+0000","last_became_active":"2024-03-16T20:34:51.909334+0000","last_became_peered":"2024-03-16T20:34:51.909334+0000","last_unstale":"2024-03-16T20:34:51.910052+0000","last_undegraded":"2024-03-16T20:34:51.910052+0000","last_fullsized":"2024-03-16T20:34:51.910052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:49:08.179435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413662+0000","last_change":"2024-03-16T20:33:48.845405+0000","last_active":"2024-03-16T20:34:51.413662+0000","last_peered":"2024-03-16T20:34:51.413662+0000","last_clean":"2024-03-16T20:34:51.413662+0000","last_became_active":"2024-03-16T20:33:48.845033+0000","last_became_peered":"2024-03-16T20:33:48.845033+0000","last_unstale":"2024-03-16T20:34:51.413662+0000","last_undegraded":"2024-03-16T20:34:51.413662+0000","last_fullsized":"2024-03-16T20:34:51.413662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:52.632960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994318+0000","last_change":"2024-03-16T20:33:49.864934+0000","last_active":"2024-03-16T20:35:19.994318+0000","last_peered":"2024-03-16T20:35:19.994318+0000","last_clean":"2024-03-16T20:35:19.994318+0000","last_became_active":"2024-03-16T20:33:49.864574+0000","last_became_peered":"2024-03-16T20:33:49.864574+0000","last_unstale":"2024-03-16T20:35:19.994318+0000","last_undegraded":"2024-03-16T20:35:19.994318+0000","last_fullsized":"2024-03-16T20:35:19.994318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:07.010788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414410+0000","last_change":"2024-03-16T20:33:53.890446+0000","last_active":"2024-03-16T20:34:51.414410+0000","last_peered":"2024-03-16T20:34:51.414410+0000","last_clean":"2024-03-16T20:34:51.414410+0000","last_became_active":"2024-03-16T20:33:53.889832+0000","last_became_peered":"2024-03-16T20:33:53.889832+0000","last_unstale":"2024-03-16T20:34:51.414410+0000","last_undegraded":"2024-03-16T20:34:51.414410+0000","last_fullsized":"2024-03-16T20:34:51.414410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:49.031999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413171+0000","last_change":"2024-03-16T20:33:48.845809+0000","last_active":"2024-03-16T20:34:51.413171+0000","last_peered":"2024-03-16T20:34:51.413171+0000","last_clean":"2024-03-16T20:34:51.413171+0000","last_became_active":"2024-03-16T20:33:48.844751+0000","last_became_peered":"2024-03-16T20:33:48.844751+0000","last_unstale":"2024-03-16T20:34:51.413171+0000","last_undegraded":"2024-03-16T20:34:51.413171+0000","last_fullsized":"2024-03-16T20:34:51.413171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:41.148435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907593+0000","last_change":"2024-03-16T20:33:49.857181+0000","last_active":"2024-03-16T20:34:51.907593+0000","last_peered":"2024-03-16T20:34:51.907593+0000","last_clean":"2024-03-16T20:34:51.907593+0000","last_became_active":"2024-03-16T20:33:49.856866+0000","last_became_peered":"2024-03-16T20:33:49.856866+0000","last_unstale":"2024-03-16T20:34:51.907593+0000","last_undegraded":"2024-03-16T20:34:51.907593+0000","last_fullsized":"2024-03-16T20:34:51.907593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.006531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419023+0000","last_change":"2024-03-16T20:33:53.879419+0000","last_active":"2024-03-16T20:34:51.419023+0000","last_peered":"2024-03-16T20:34:51.419023+0000","last_clean":"2024-03-16T20:34:51.419023+0000","last_became_active":"2024-03-16T20:33:53.878590+0000","last_became_peered":"2024-03-16T20:33:53.878590+0000","last_unstale":"2024-03-16T20:34:51.419023+0000","last_undegraded":"2024-03-16T20:34:51.419023+0000","last_fullsized":"2024-03-16T20:34:51.419023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:06:34.552834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:17.998840+0000","last_change":"2024-03-16T20:33:48.845122+0000","last_active":"2024-03-16T20:35:17.998840+0000","last_peered":"2024-03-16T20:35:17.998840+0000","last_clean":"2024-03-16T20:35:17.998840+0000","last_became_active":"2024-03-16T20:33:48.844109+0000","last_became_peered":"2024-03-16T20:33:48.844109+0000","last_unstale":"2024-03-16T20:35:17.998840+0000","last_undegraded":"2024-03-16T20:35:17.998840+0000","last_fullsized":"2024-03-16T20:35:17.998840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:46.911439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.957672+0000","last_change":"2024-03-16T20:33:49.850398+0000","last_active":"2024-03-16T20:35:19.957672+0000","last_peered":"2024-03-16T20:35:19.957672+0000","last_clean":"2024-03-16T20:35:19.957672+0000","last_became_active":"2024-03-16T20:33:49.850059+0000","last_became_peered":"2024-03-16T20:33:49.850059+0000","last_unstale":"2024-03-16T20:35:19.957672+0000","last_undegraded":"2024-03-16T20:35:19.957672+0000","last_fullsized":"2024-03-16T20:35:19.957672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:02.193759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419876+0000","last_change":"2024-03-16T20:33:53.890035+0000","last_active":"2024-03-16T20:34:51.419876+0000","last_peered":"2024-03-16T20:34:51.419876+0000","last_clean":"2024-03-16T20:34:51.419876+0000","last_became_active":"2024-03-16T20:33:53.889468+0000","last_became_peered":"2024-03-16T20:33:53.889468+0000","last_unstale":"2024-03-16T20:34:51.419876+0000","last_undegraded":"2024-03-16T20:34:51.419876+0000","last_fullsized":"2024-03-16T20:34:51.419876+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:47.125880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.760192+0000","last_change":"2024-03-16T20:33:48.850165+0000","last_active":"2024-03-16T20:34:51.760192+0000","last_peered":"2024-03-16T20:34:51.760192+0000","last_clean":"2024-03-16T20:34:51.760192+0000","last_became_active":"2024-03-16T20:33:48.849826+0000","last_became_peered":"2024-03-16T20:33:48.849826+0000","last_unstale":"2024-03-16T20:34:51.760192+0000","last_undegraded":"2024-03-16T20:34:51.760192+0000","last_fullsized":"2024-03-16T20:34:51.760192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:23.790227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907632+0000","last_change":"2024-03-16T20:33:49.857506+0000","last_active":"2024-03-16T20:34:51.907632+0000","last_peered":"2024-03-16T20:34:51.907632+0000","last_clean":"2024-03-16T20:34:51.907632+0000","last_became_active":"2024-03-16T20:33:49.856343+0000","last_became_peered":"2024-03-16T20:33:49.856343+0000","last_unstale":"2024-03-16T20:34:51.907632+0000","last_undegraded":"2024-03-16T20:34:51.907632+0000","last_fullsized":"2024-03-16T20:34:51.907632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:18:02.747518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.420360+0000","last_change":"2024-03-16T20:34:51.420360+0000","last_active":"2024-03-16T20:34:51.420360+0000","last_peered":"2024-03-16T20:34:51.420360+0000","last_clean":"2024-03-16T20:34:51.420360+0000","last_became_active":"2024-03-16T20:34:51.419906+0000","last_became_peered":"2024-03-16T20:34:51.419906+0000","last_unstale":"2024-03-16T20:34:51.420360+0000","last_undegraded":"2024-03-16T20:34:51.420360+0000","last_fullsized":"2024-03-16T20:34:51.420360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:47.903906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415964+0000","last_change":"2024-03-16T20:33:48.849773+0000","last_active":"2024-03-16T20:34:51.415964+0000","last_peered":"2024-03-16T20:34:51.415964+0000","last_clean":"2024-03-16T20:34:51.415964+0000","last_became_active":"2024-03-16T20:33:48.848736+0000","last_became_peered":"2024-03-16T20:33:48.848736+0000","last_unstale":"2024-03-16T20:34:51.415964+0000","last_undegraded":"2024-03-16T20:34:51.415964+0000","last_fullsized":"2024-03-16T20:34:51.415964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:54:37.017119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983450+0000","last_change":"2024-03-16T20:33:49.853446+0000","last_active":"2024-03-16T20:35:19.983450+0000","last_peered":"2024-03-16T20:35:19.983450+0000","last_clean":"2024-03-16T20:35:19.983450+0000","last_became_active":"2024-03-16T20:33:49.851886+0000","last_became_peered":"2024-03-16T20:33:49.851886+0000","last_unstale":"2024-03-16T20:35:19.983450+0000","last_undegraded":"2024-03-16T20:35:19.983450+0000","last_fullsized":"2024-03-16T20:35:19.983450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:39.675229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414121+0000","last_change":"2024-03-16T20:33:53.890388+0000","last_active":"2024-03-16T20:34:51.414121+0000","last_peered":"2024-03-16T20:34:51.414121+0000","last_clean":"2024-03-16T20:34:51.414121+0000","last_became_active":"2024-03-16T20:33:53.889966+0000","last_became_peered":"2024-03-16T20:33:53.889966+0000","last_unstale":"2024-03-16T20:34:51.414121+0000","last_undegraded":"2024-03-16T20:34:51.414121+0000","last_fullsized":"2024-03-16T20:34:51.414121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:21.475392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907286+0000","last_change":"2024-03-16T20:33:48.841524+0000","last_active":"2024-03-16T20:34:51.907286+0000","last_peered":"2024-03-16T20:34:51.907286+0000","last_clean":"2024-03-16T20:34:51.907286+0000","last_became_active":"2024-03-16T20:33:48.841239+0000","last_became_peered":"2024-03-16T20:33:48.841239+0000","last_unstale":"2024-03-16T20:34:51.907286+0000","last_undegraded":"2024-03-16T20:34:51.907286+0000","last_fullsized":"2024-03-16T20:34:51.907286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:42.206113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979408+0000","last_change":"2024-03-16T20:33:49.856131+0000","last_active":"2024-03-16T20:35:19.979408+0000","last_peered":"2024-03-16T20:35:19.979408+0000","last_clean":"2024-03-16T20:35:19.979408+0000","last_became_active":"2024-03-16T20:33:49.855819+0000","last_became_peered":"2024-03-16T20:33:49.855819+0000","last_unstale":"2024-03-16T20:35:19.979408+0000","last_undegraded":"2024-03-16T20:35:19.979408+0000","last_fullsized":"2024-03-16T20:35:19.979408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:22.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417444+0000","last_change":"2024-03-16T20:33:53.888159+0000","last_active":"2024-03-16T20:34:51.417444+0000","last_peered":"2024-03-16T20:34:51.417444+0000","last_clean":"2024-03-16T20:34:51.417444+0000","last_became_active":"2024-03-16T20:33:53.887777+0000","last_became_peered":"2024-03-16T20:33:53.887777+0000","last_unstale":"2024-03-16T20:34:51.417444+0000","last_undegraded":"2024-03-16T20:34:51.417444+0000","last_fullsized":"2024-03-16T20:34:51.417444+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:26:17.021957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":176,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:21.001698+0000","last_change":"2024-03-16T20:33:48.846649+0000","last_active":"2024-03-16T20:35:21.001698+0000","last_peered":"2024-03-16T20:35:21.001698+0000","last_clean":"2024-03-16T20:35:21.001698+0000","last_became_active":"2024-03-16T20:33:48.846344+0000","last_became_peered":"2024-03-16T20:33:48.846344+0000","last_unstale":"2024-03-16T20:35:21.001698+0000","last_undegraded":"2024-03-16T20:35:21.001698+0000","last_fullsized":"2024-03-16T20:35:21.001698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:54:24.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994181+0000","last_change":"2024-03-16T20:33:49.866087+0000","last_active":"2024-03-16T20:35:19.994181+0000","last_peered":"2024-03-16T20:35:19.994181+0000","last_clean":"2024-03-16T20:35:19.994181+0000","last_became_active":"2024-03-16T20:33:49.865722+0000","last_became_peered":"2024-03-16T20:33:49.865722+0000","last_unstale":"2024-03-16T20:35:19.994181+0000","last_undegraded":"2024-03-16T20:35:19.994181+0000","last_fullsized":"2024-03-16T20:35:19.994181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:04:09.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412158+0000","last_change":"2024-03-16T20:33:53.890208+0000","last_active":"2024-03-16T20:34:51.412158+0000","last_peered":"2024-03-16T20:34:51.412158+0000","last_clean":"2024-03-16T20:34:51.412158+0000","last_became_active":"2024-03-16T20:33:53.889807+0000","last_became_peered":"2024-03-16T20:33:53.889807+0000","last_unstale":"2024-03-16T20:34:51.412158+0000","last_undegraded":"2024-03-16T20:34:51.412158+0000","last_fullsized":"2024-03-16T20:34:51.412158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:30:53.439687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418606+0000","last_change":"2024-03-16T20:33:48.851610+0000","last_active":"2024-03-16T20:34:51.418606+0000","last_peered":"2024-03-16T20:34:51.418606+0000","last_clean":"2024-03-16T20:34:51.418606+0000","last_became_active":"2024-03-16T20:33:48.851054+0000","last_became_peered":"2024-03-16T20:33:48.851054+0000","last_unstale":"2024-03-16T20:34:51.418606+0000","last_undegraded":"2024-03-16T20:34:51.418606+0000","last_fullsized":"2024-03-16T20:34:51.418606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:58:03.733318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414439+0000","last_change":"2024-03-16T20:33:49.856534+0000","last_active":"2024-03-16T20:34:51.414439+0000","last_peered":"2024-03-16T20:34:51.414439+0000","last_clean":"2024-03-16T20:34:51.414439+0000","last_became_active":"2024-03-16T20:33:49.855957+0000","last_became_peered":"2024-03-16T20:33:49.855957+0000","last_unstale":"2024-03-16T20:34:51.414439+0000","last_undegraded":"2024-03-16T20:34:51.414439+0000","last_fullsized":"2024-03-16T20:34:51.414439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:07.035334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415269+0000","last_change":"2024-03-16T20:33:53.892149+0000","last_active":"2024-03-16T20:34:51.415269+0000","last_peered":"2024-03-16T20:34:51.415269+0000","last_clean":"2024-03-16T20:34:51.415269+0000","last_became_active":"2024-03-16T20:33:53.891730+0000","last_became_peered":"2024-03-16T20:33:53.891730+0000","last_unstale":"2024-03-16T20:34:51.415269+0000","last_undegraded":"2024-03-16T20:34:51.415269+0000","last_fullsized":"2024-03-16T20:34:51.415269+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:59.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907483+0000","last_change":"2024-03-16T20:33:48.849909+0000","last_active":"2024-03-16T20:34:51.907483+0000","last_peered":"2024-03-16T20:34:51.907483+0000","last_clean":"2024-03-16T20:34:51.907483+0000","last_became_active":"2024-03-16T20:33:48.848645+0000","last_became_peered":"2024-03-16T20:33:48.848645+0000","last_unstale":"2024-03-16T20:34:51.907483+0000","last_undegraded":"2024-03-16T20:34:51.907483+0000","last_fullsized":"2024-03-16T20:34:51.907483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:44.401679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.992869+0000","last_change":"2024-03-16T20:33:49.853100+0000","last_active":"2024-03-16T20:35:19.992869+0000","last_peered":"2024-03-16T20:35:19.992869+0000","last_clean":"2024-03-16T20:35:19.992869+0000","last_became_active":"2024-03-16T20:33:49.852743+0000","last_became_peered":"2024-03-16T20:33:49.852743+0000","last_unstale":"2024-03-16T20:35:19.992869+0000","last_undegraded":"2024-03-16T20:35:19.992869+0000","last_fullsized":"2024-03-16T20:35:19.992869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:42.664026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412319+0000","last_change":"2024-03-16T20:33:53.881626+0000","last_active":"2024-03-16T20:34:51.412319+0000","last_peered":"2024-03-16T20:34:51.412319+0000","last_clean":"2024-03-16T20:34:51.412319+0000","last_became_active":"2024-03-16T20:33:53.881205+0000","last_became_peered":"2024-03-16T20:33:53.881205+0000","last_unstale":"2024-03-16T20:34:51.412319+0000","last_undegraded":"2024-03-16T20:34:51.412319+0000","last_fullsized":"2024-03-16T20:34:51.412319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:19.525953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'35","reported_seq":217,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:21.004380+0000","last_change":"2024-03-16T20:33:48.848490+0000","last_active":"2024-03-16T20:35:21.004380+0000","last_peered":"2024-03-16T20:35:21.004380+0000","last_clean":"2024-03-16T20:35:21.004380+0000","last_became_active":"2024-03-16T20:33:48.848152+0000","last_became_peered":"2024-03-16T20:33:48.848152+0000","last_unstale":"2024-03-16T20:35:21.004380+0000","last_undegraded":"2024-03-16T20:35:21.004380+0000","last_fullsized":"2024-03-16T20:35:21.004380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:43:19.908178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":155,"num_read_kb":138,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.955150+0000","last_change":"2024-03-16T20:33:49.851269+0000","last_active":"2024-03-16T20:35:19.955150+0000","last_peered":"2024-03-16T20:35:19.955150+0000","last_clean":"2024-03-16T20:35:19.955150+0000","last_became_active":"2024-03-16T20:33:49.850963+0000","last_became_peered":"2024-03-16T20:33:49.850963+0000","last_unstale":"2024-03-16T20:35:19.955150+0000","last_undegraded":"2024-03-16T20:35:19.955150+0000","last_fullsized":"2024-03-16T20:35:19.955150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:26:01.312043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907845+0000","last_change":"2024-03-16T20:33:53.889419+0000","last_active":"2024-03-16T20:34:51.907845+0000","last_peered":"2024-03-16T20:34:51.907845+0000","last_clean":"2024-03-16T20:34:51.907845+0000","last_became_active":"2024-03-16T20:33:53.889016+0000","last_became_peered":"2024-03-16T20:33:53.889016+0000","last_unstale":"2024-03-16T20:34:51.907845+0000","last_undegraded":"2024-03-16T20:34:51.907845+0000","last_fullsized":"2024-03-16T20:34:51.907845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:32:58.333441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419922+0000","last_change":"2024-03-16T20:33:48.855070+0000","last_active":"2024-03-16T20:34:51.419922+0000","last_peered":"2024-03-16T20:34:51.419922+0000","last_clean":"2024-03-16T20:34:51.419922+0000","last_became_active":"2024-03-16T20:33:48.854669+0000","last_became_peered":"2024-03-16T20:33:48.854669+0000","last_unstale":"2024-03-16T20:34:51.419922+0000","last_undegraded":"2024-03-16T20:34:51.419922+0000","last_fullsized":"2024-03-16T20:34:51.419922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:38.229641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983523+0000","last_change":"2024-03-16T20:33:49.850186+0000","last_active":"2024-03-16T20:35:19.983523+0000","last_peered":"2024-03-16T20:35:19.983523+0000","last_clean":"2024-03-16T20:35:19.983523+0000","last_became_active":"2024-03-16T20:33:49.849811+0000","last_became_peered":"2024-03-16T20:33:49.849811+0000","last_unstale":"2024-03-16T20:35:19.983523+0000","last_undegraded":"2024-03-16T20:35:19.983523+0000","last_fullsized":"2024-03-16T20:35:19.983523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:13:03.399984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415127+0000","last_change":"2024-03-16T20:33:53.890030+0000","last_active":"2024-03-16T20:34:51.415127+0000","last_peered":"2024-03-16T20:34:51.415127+0000","last_clean":"2024-03-16T20:34:51.415127+0000","last_became_active":"2024-03-16T20:33:53.889602+0000","last_became_peered":"2024-03-16T20:33:53.889602+0000","last_unstale":"2024-03-16T20:34:51.415127+0000","last_undegraded":"2024-03-16T20:34:51.415127+0000","last_fullsized":"2024-03-16T20:34:51.415127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:04.319150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907456+0000","last_change":"2024-03-16T20:33:48.840422+0000","last_active":"2024-03-16T20:34:51.907456+0000","last_peered":"2024-03-16T20:34:51.907456+0000","last_clean":"2024-03-16T20:34:51.907456+0000","last_became_active":"2024-03-16T20:33:48.840125+0000","last_became_peered":"2024-03-16T20:33:48.840125+0000","last_unstale":"2024-03-16T20:34:51.907456+0000","last_undegraded":"2024-03-16T20:34:51.907456+0000","last_fullsized":"2024-03-16T20:34:51.907456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:05.397252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418151+0000","last_change":"2024-03-16T20:33:49.862475+0000","last_active":"2024-03-16T20:34:51.418151+0000","last_peered":"2024-03-16T20:34:51.418151+0000","last_clean":"2024-03-16T20:34:51.418151+0000","last_became_active":"2024-03-16T20:33:49.862032+0000","last_became_peered":"2024-03-16T20:33:49.862032+0000","last_unstale":"2024-03-16T20:34:51.418151+0000","last_undegraded":"2024-03-16T20:34:51.418151+0000","last_fullsized":"2024-03-16T20:34:51.418151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:16:53.319352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412513+0000","last_change":"2024-03-16T20:33:53.884019+0000","last_active":"2024-03-16T20:34:51.412513+0000","last_peered":"2024-03-16T20:34:51.412513+0000","last_clean":"2024-03-16T20:34:51.412513+0000","last_became_active":"2024-03-16T20:33:53.883608+0000","last_became_peered":"2024-03-16T20:33:53.883608+0000","last_unstale":"2024-03-16T20:34:51.412513+0000","last_undegraded":"2024-03-16T20:34:51.412513+0000","last_fullsized":"2024-03-16T20:34:51.412513+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:58.689682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":105,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:18.002758+0000","last_change":"2024-03-16T20:33:48.848145+0000","last_active":"2024-03-16T20:35:18.002758+0000","last_peered":"2024-03-16T20:35:18.002758+0000","last_clean":"2024-03-16T20:35:18.002758+0000","last_became_active":"2024-03-16T20:33:48.847754+0000","last_became_peered":"2024-03-16T20:33:48.847754+0000","last_unstale":"2024-03-16T20:35:18.002758+0000","last_undegraded":"2024-03-16T20:35:18.002758+0000","last_fullsized":"2024-03-16T20:35:18.002758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:59.967346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417966+0000","last_change":"2024-03-16T20:33:49.851307+0000","last_active":"2024-03-16T20:34:51.417966+0000","last_peered":"2024-03-16T20:34:51.417966+0000","last_clean":"2024-03-16T20:34:51.417966+0000","last_became_active":"2024-03-16T20:33:49.850930+0000","last_became_peered":"2024-03-16T20:33:49.850930+0000","last_unstale":"2024-03-16T20:34:51.417966+0000","last_undegraded":"2024-03-16T20:34:51.417966+0000","last_fullsized":"2024-03-16T20:34:51.417966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:23:54.528895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416343+0000","last_change":"2024-03-16T20:33:53.893149+0000","last_active":"2024-03-16T20:34:51.416343+0000","last_peered":"2024-03-16T20:34:51.416343+0000","last_clean":"2024-03-16T20:34:51.416343+0000","last_became_active":"2024-03-16T20:33:53.892532+0000","last_became_peered":"2024-03-16T20:33:53.892532+0000","last_unstale":"2024-03-16T20:34:51.416343+0000","last_undegraded":"2024-03-16T20:34:51.416343+0000","last_fullsized":"2024-03-16T20:34:51.416343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:09.408974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412677+0000","last_change":"2024-03-16T20:33:48.849788+0000","last_active":"2024-03-16T20:34:51.412677+0000","last_peered":"2024-03-16T20:34:51.412677+0000","last_clean":"2024-03-16T20:34:51.412677+0000","last_became_active":"2024-03-16T20:33:48.849360+0000","last_became_peered":"2024-03-16T20:33:48.849360+0000","last_unstale":"2024-03-16T20:34:51.412677+0000","last_undegraded":"2024-03-16T20:34:51.412677+0000","last_fullsized":"2024-03-16T20:34:51.412677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:29.026216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907448+0000","last_change":"2024-03-16T20:33:49.856702+0000","last_active":"2024-03-16T20:34:51.907448+0000","last_peered":"2024-03-16T20:34:51.907448+0000","last_clean":"2024-03-16T20:34:51.907448+0000","last_became_active":"2024-03-16T20:33:49.856373+0000","last_became_peered":"2024-03-16T20:33:49.856373+0000","last_unstale":"2024-03-16T20:34:51.907448+0000","last_undegraded":"2024-03-16T20:34:51.907448+0000","last_fullsized":"2024-03-16T20:34:51.907448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:31:40.093440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907350+0000","last_change":"2024-03-16T20:33:53.884193+0000","last_active":"2024-03-16T20:34:51.907350+0000","last_peered":"2024-03-16T20:34:51.907350+0000","last_clean":"2024-03-16T20:34:51.907350+0000","last_became_active":"2024-03-16T20:33:53.883763+0000","last_became_peered":"2024-03-16T20:33:53.883763+0000","last_unstale":"2024-03-16T20:34:51.907350+0000","last_undegraded":"2024-03-16T20:34:51.907350+0000","last_fullsized":"2024-03-16T20:34:51.907350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:50:20.969662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413379+0000","last_change":"2024-03-16T20:33:48.851166+0000","last_active":"2024-03-16T20:34:51.413379+0000","last_peered":"2024-03-16T20:34:51.413379+0000","last_clean":"2024-03-16T20:34:51.413379+0000","last_became_active":"2024-03-16T20:33:48.850843+0000","last_became_peered":"2024-03-16T20:33:48.850843+0000","last_unstale":"2024-03-16T20:34:51.413379+0000","last_undegraded":"2024-03-16T20:34:51.413379+0000","last_fullsized":"2024-03-16T20:34:51.413379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:09.142942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415100+0000","last_change":"2024-03-16T20:33:49.860038+0000","last_active":"2024-03-16T20:34:51.415100+0000","last_peered":"2024-03-16T20:34:51.415100+0000","last_clean":"2024-03-16T20:34:51.415100+0000","last_became_active":"2024-03-16T20:33:49.859715+0000","last_became_peered":"2024-03-16T20:33:49.859715+0000","last_unstale":"2024-03-16T20:34:51.415100+0000","last_undegraded":"2024-03-16T20:34:51.415100+0000","last_fullsized":"2024-03-16T20:34:51.415100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:27:32.800166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413095+0000","last_change":"2024-03-16T20:33:53.930129+0000","last_active":"2024-03-16T20:34:51.413095+0000","last_peered":"2024-03-16T20:34:51.413095+0000","last_clean":"2024-03-16T20:34:51.413095+0000","last_became_active":"2024-03-16T20:33:53.929655+0000","last_became_peered":"2024-03-16T20:33:53.929655+0000","last_unstale":"2024-03-16T20:34:51.413095+0000","last_undegraded":"2024-03-16T20:34:51.413095+0000","last_fullsized":"2024-03-16T20:34:51.413095+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:46.422720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412652+0000","last_change":"2024-03-16T20:33:48.849089+0000","last_active":"2024-03-16T20:34:51.412652+0000","last_peered":"2024-03-16T20:34:51.412652+0000","last_clean":"2024-03-16T20:34:51.412652+0000","last_became_active":"2024-03-16T20:33:48.848719+0000","last_became_peered":"2024-03-16T20:33:48.848719+0000","last_unstale":"2024-03-16T20:34:51.412652+0000","last_undegraded":"2024-03-16T20:34:51.412652+0000","last_fullsized":"2024-03-16T20:34:51.412652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:31:23.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.962814+0000","last_change":"2024-03-16T20:33:49.853247+0000","last_active":"2024-03-16T20:35:19.962814+0000","last_peered":"2024-03-16T20:35:19.962814+0000","last_clean":"2024-03-16T20:35:19.962814+0000","last_became_active":"2024-03-16T20:33:49.852870+0000","last_became_peered":"2024-03-16T20:33:49.852870+0000","last_unstale":"2024-03-16T20:35:19.962814+0000","last_undegraded":"2024-03-16T20:35:19.962814+0000","last_fullsized":"2024-03-16T20:35:19.962814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:56.365484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416383+0000","last_change":"2024-03-16T20:33:53.887350+0000","last_active":"2024-03-16T20:34:51.416383+0000","last_peered":"2024-03-16T20:34:51.416383+0000","last_clean":"2024-03-16T20:34:51.416383+0000","last_became_active":"2024-03-16T20:33:53.886782+0000","last_became_peered":"2024-03-16T20:33:53.886782+0000","last_unstale":"2024-03-16T20:34:51.416383+0000","last_undegraded":"2024-03-16T20:34:51.416383+0000","last_fullsized":"2024-03-16T20:34:51.416383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:00:39.898430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":102,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:18.035444+0000","last_change":"2024-03-16T20:33:48.851543+0000","last_active":"2024-03-16T20:35:18.035444+0000","last_peered":"2024-03-16T20:35:18.035444+0000","last_clean":"2024-03-16T20:35:18.035444+0000","last_became_active":"2024-03-16T20:33:48.851196+0000","last_became_peered":"2024-03-16T20:33:48.851196+0000","last_unstale":"2024-03-16T20:35:18.035444+0000","last_undegraded":"2024-03-16T20:35:18.035444+0000","last_fullsized":"2024-03-16T20:35:18.035444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:06.309259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.962510+0000","last_change":"2024-03-16T20:33:49.856021+0000","last_active":"2024-03-16T20:35:19.962510+0000","last_peered":"2024-03-16T20:35:19.962510+0000","last_clean":"2024-03-16T20:35:19.962510+0000","last_became_active":"2024-03-16T20:33:49.854130+0000","last_became_peered":"2024-03-16T20:33:49.854130+0000","last_unstale":"2024-03-16T20:35:19.962510+0000","last_undegraded":"2024-03-16T20:35:19.962510+0000","last_fullsized":"2024-03-16T20:35:19.962510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:29:39.970391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416561+0000","last_change":"2024-03-16T20:33:53.930094+0000","last_active":"2024-03-16T20:34:51.416561+0000","last_peered":"2024-03-16T20:34:51.416561+0000","last_clean":"2024-03-16T20:34:51.416561+0000","last_became_active":"2024-03-16T20:33:53.929613+0000","last_became_peered":"2024-03-16T20:33:53.929613+0000","last_unstale":"2024-03-16T20:34:51.416561+0000","last_undegraded":"2024-03-16T20:34:51.416561+0000","last_fullsized":"2024-03-16T20:34:51.416561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:30.680295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907471+0000","last_change":"2024-03-16T20:33:49.857949+0000","last_active":"2024-03-16T20:34:51.907471+0000","last_peered":"2024-03-16T20:34:51.907471+0000","last_clean":"2024-03-16T20:34:51.907471+0000","last_became_active":"2024-03-16T20:33:49.857642+0000","last_became_peered":"2024-03-16T20:33:49.857642+0000","last_unstale":"2024-03-16T20:34:51.907471+0000","last_undegraded":"2024-03-16T20:34:51.907471+0000","last_fullsized":"2024-03-16T20:34:51.907471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:23.040787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":105,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.003384+0000","last_change":"2024-03-16T20:33:48.856908+0000","last_active":"2024-03-16T20:35:19.003384+0000","last_peered":"2024-03-16T20:35:19.003384+0000","last_clean":"2024-03-16T20:35:19.003384+0000","last_became_active":"2024-03-16T20:33:48.856578+0000","last_became_peered":"2024-03-16T20:33:48.856578+0000","last_unstale":"2024-03-16T20:35:19.003384+0000","last_undegraded":"2024-03-16T20:35:19.003384+0000","last_fullsized":"2024-03-16T20:35:19.003384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:37.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418818+0000","last_change":"2024-03-16T20:33:53.881965+0000","last_active":"2024-03-16T20:34:51.418818+0000","last_peered":"2024-03-16T20:34:51.418818+0000","last_clean":"2024-03-16T20:34:51.418818+0000","last_became_active":"2024-03-16T20:33:53.881247+0000","last_became_peered":"2024-03-16T20:33:53.881247+0000","last_unstale":"2024-03-16T20:34:51.418818+0000","last_undegraded":"2024-03-16T20:34:51.418818+0000","last_fullsized":"2024-03-16T20:34:51.418818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:18.868143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.957596+0000","last_change":"2024-03-16T20:34:51.909039+0000","last_active":"2024-03-16T20:35:19.957596+0000","last_peered":"2024-03-16T20:35:19.957596+0000","last_clean":"2024-03-16T20:35:19.957596+0000","last_became_active":"2024-03-16T20:34:51.908628+0000","last_became_peered":"2024-03-16T20:34:51.908628+0000","last_unstale":"2024-03-16T20:35:19.957596+0000","last_undegraded":"2024-03-16T20:35:19.957596+0000","last_fullsized":"2024-03-16T20:35:19.957596+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:02.007925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415231+0000","last_change":"2024-03-16T20:33:48.850177+0000","last_active":"2024-03-16T20:34:51.415231+0000","last_peered":"2024-03-16T20:34:51.415231+0000","last_clean":"2024-03-16T20:34:51.415231+0000","last_became_active":"2024-03-16T20:33:48.849838+0000","last_became_peered":"2024-03-16T20:33:48.849838+0000","last_unstale":"2024-03-16T20:34:51.415231+0000","last_undegraded":"2024-03-16T20:34:51.415231+0000","last_fullsized":"2024-03-16T20:34:51.415231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:24.544880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417808+0000","last_change":"2024-03-16T20:32:34.913833+0000","last_active":"2024-03-16T20:34:51.417808+0000","last_peered":"2024-03-16T20:34:51.417808+0000","last_clean":"2024-03-16T20:34:51.417808+0000","last_became_active":"2024-03-16T20:32:34.913443+0000","last_became_peered":"2024-03-16T20:32:34.913443+0000","last_unstale":"2024-03-16T20:34:51.417808+0000","last_undegraded":"2024-03-16T20:34:51.417808+0000","last_fullsized":"2024-03-16T20:34:51.417808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_clean_scrub_stamp":"2024-03-16T20:29:10.760383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:05.333386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907623+0000","last_change":"2024-03-16T20:33:53.931311+0000","last_active":"2024-03-16T20:34:51.907623+0000","last_peered":"2024-03-16T20:34:51.907623+0000","last_clean":"2024-03-16T20:34:51.907623+0000","last_became_active":"2024-03-16T20:33:53.930972+0000","last_became_peered":"2024-03-16T20:33:53.930972+0000","last_unstale":"2024-03-16T20:34:51.907623+0000","last_undegraded":"2024-03-16T20:34:51.907623+0000","last_fullsized":"2024-03-16T20:34:51.907623+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:29.776651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.981100+0000","last_change":"2024-03-16T20:33:49.838866+0000","last_active":"2024-03-16T20:35:19.981100+0000","last_peered":"2024-03-16T20:35:19.981100+0000","last_clean":"2024-03-16T20:35:19.981100+0000","last_became_active":"2024-03-16T20:33:49.838561+0000","last_became_peered":"2024-03-16T20:33:49.838561+0000","last_unstale":"2024-03-16T20:35:19.981100+0000","last_undegraded":"2024-03-16T20:35:19.981100+0000","last_fullsized":"2024-03-16T20:35:19.981100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:57:21.098078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417865+0000","last_change":"2024-03-16T20:33:48.847098+0000","last_active":"2024-03-16T20:34:51.417865+0000","last_peered":"2024-03-16T20:34:51.417865+0000","last_clean":"2024-03-16T20:34:51.417865+0000","last_became_active":"2024-03-16T20:33:48.846399+0000","last_became_peered":"2024-03-16T20:33:48.846399+0000","last_unstale":"2024-03-16T20:34:51.417865+0000","last_undegraded":"2024-03-16T20:34:51.417865+0000","last_fullsized":"2024-03-16T20:34:51.417865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:59.126547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415394+0000","last_change":"2024-03-16T20:33:53.890952+0000","last_active":"2024-03-16T20:34:51.415394+0000","last_peered":"2024-03-16T20:34:51.415394+0000","last_clean":"2024-03-16T20:34:51.415394+0000","last_became_active":"2024-03-16T20:33:53.890473+0000","last_became_peered":"2024-03-16T20:33:53.890473+0000","last_unstale":"2024-03-16T20:34:51.415394+0000","last_undegraded":"2024-03-16T20:34:51.415394+0000","last_fullsized":"2024-03-16T20:34:51.415394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:06.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413602+0000","last_change":"2024-03-16T20:33:48.851220+0000","last_active":"2024-03-16T20:34:51.413602+0000","last_peered":"2024-03-16T20:34:51.413602+0000","last_clean":"2024-03-16T20:34:51.413602+0000","last_became_active":"2024-03-16T20:33:48.850927+0000","last_became_peered":"2024-03-16T20:33:48.850927+0000","last_unstale":"2024-03-16T20:34:51.413602+0000","last_undegraded":"2024-03-16T20:34:51.413602+0000","last_fullsized":"2024-03-16T20:34:51.413602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:50.105276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983955+0000","last_change":"2024-03-16T20:33:49.850272+0000","last_active":"2024-03-16T20:35:19.983955+0000","last_peered":"2024-03-16T20:35:19.983955+0000","last_clean":"2024-03-16T20:35:19.983955+0000","last_became_active":"2024-03-16T20:33:49.849913+0000","last_became_peered":"2024-03-16T20:33:49.849913+0000","last_unstale":"2024-03-16T20:35:19.983955+0000","last_undegraded":"2024-03-16T20:35:19.983955+0000","last_fullsized":"2024-03-16T20:35:19.983955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:53:18.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417469+0000","last_change":"2024-03-16T20:33:53.930421+0000","last_active":"2024-03-16T20:34:51.417469+0000","last_peered":"2024-03-16T20:34:51.417469+0000","last_clean":"2024-03-16T20:34:51.417469+0000","last_became_active":"2024-03-16T20:33:53.929957+0000","last_became_peered":"2024-03-16T20:33:53.929957+0000","last_unstale":"2024-03-16T20:34:51.417469+0000","last_undegraded":"2024-03-16T20:34:51.417469+0000","last_fullsized":"2024-03-16T20:34:51.417469+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:20.541592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412961+0000","last_change":"2024-03-16T20:33:48.845096+0000","last_active":"2024-03-16T20:34:51.412961+0000","last_peered":"2024-03-16T20:34:51.412961+0000","last_clean":"2024-03-16T20:34:51.412961+0000","last_became_active":"2024-03-16T20:33:48.844580+0000","last_became_peered":"2024-03-16T20:33:48.844580+0000","last_unstale":"2024-03-16T20:34:51.412961+0000","last_undegraded":"2024-03-16T20:34:51.412961+0000","last_fullsized":"2024-03-16T20:34:51.412961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:56:53.857336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.980180+0000","last_change":"2024-03-16T20:33:49.858488+0000","last_active":"2024-03-16T20:35:19.980180+0000","last_peered":"2024-03-16T20:35:19.980180+0000","last_clean":"2024-03-16T20:35:19.980180+0000","last_became_active":"2024-03-16T20:33:49.858163+0000","last_became_peered":"2024-03-16T20:33:49.858163+0000","last_unstale":"2024-03-16T20:35:19.980180+0000","last_undegraded":"2024-03-16T20:35:19.980180+0000","last_fullsized":"2024-03-16T20:35:19.980180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:49.399329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417853+0000","last_change":"2024-03-16T20:33:53.884240+0000","last_active":"2024-03-16T20:34:51.417853+0000","last_peered":"2024-03-16T20:34:51.417853+0000","last_clean":"2024-03-16T20:34:51.417853+0000","last_became_active":"2024-03-16T20:33:53.883864+0000","last_became_peered":"2024-03-16T20:33:53.883864+0000","last_unstale":"2024-03-16T20:34:51.417853+0000","last_undegraded":"2024-03-16T20:34:51.417853+0000","last_fullsized":"2024-03-16T20:34:51.417853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:57:56.073112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415623+0000","last_change":"2024-03-16T20:33:48.858217+0000","last_active":"2024-03-16T20:34:51.415623+0000","last_peered":"2024-03-16T20:34:51.415623+0000","last_clean":"2024-03-16T20:34:51.415623+0000","last_became_active":"2024-03-16T20:33:48.857844+0000","last_became_peered":"2024-03-16T20:33:48.857844+0000","last_unstale":"2024-03-16T20:34:51.415623+0000","last_undegraded":"2024-03-16T20:34:51.415623+0000","last_fullsized":"2024-03-16T20:34:51.415623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:09.143250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414570+0000","last_change":"2024-03-16T20:33:49.856002+0000","last_active":"2024-03-16T20:34:51.414570+0000","last_peered":"2024-03-16T20:34:51.414570+0000","last_clean":"2024-03-16T20:34:51.414570+0000","last_became_active":"2024-03-16T20:33:49.855664+0000","last_became_peered":"2024-03-16T20:33:49.855664+0000","last_unstale":"2024-03-16T20:34:51.414570+0000","last_undegraded":"2024-03-16T20:34:51.414570+0000","last_fullsized":"2024-03-16T20:34:51.414570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:55:36.502729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.761360+0000","last_change":"2024-03-16T20:34:51.761360+0000","last_active":"2024-03-16T20:34:51.761360+0000","last_peered":"2024-03-16T20:34:51.761360+0000","last_clean":"2024-03-16T20:34:51.761360+0000","last_became_active":"2024-03-16T20:34:51.760903+0000","last_became_peered":"2024-03-16T20:34:51.760903+0000","last_unstale":"2024-03-16T20:34:51.761360+0000","last_undegraded":"2024-03-16T20:34:51.761360+0000","last_fullsized":"2024-03-16T20:34:51.761360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:08.015778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907519+0000","last_change":"2024-03-16T20:33:48.855853+0000","last_active":"2024-03-16T20:34:51.907519+0000","last_peered":"2024-03-16T20:34:51.907519+0000","last_clean":"2024-03-16T20:34:51.907519+0000","last_became_active":"2024-03-16T20:33:48.855539+0000","last_became_peered":"2024-03-16T20:33:48.855539+0000","last_unstale":"2024-03-16T20:34:51.907519+0000","last_undegraded":"2024-03-16T20:34:51.907519+0000","last_fullsized":"2024-03-16T20:34:51.907519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:56:48.618350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.935552+0000","last_change":"2024-03-16T20:33:49.854825+0000","last_active":"2024-03-16T20:35:19.935552+0000","last_peered":"2024-03-16T20:35:19.935552+0000","last_clean":"2024-03-16T20:35:19.935552+0000","last_became_active":"2024-03-16T20:33:49.854396+0000","last_became_peered":"2024-03-16T20:33:49.854396+0000","last_unstale":"2024-03-16T20:35:19.935552+0000","last_undegraded":"2024-03-16T20:35:19.935552+0000","last_fullsized":"2024-03-16T20:35:19.935552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:31:18.133694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412944+0000","last_change":"2024-03-16T20:33:53.881554+0000","last_active":"2024-03-16T20:34:51.412944+0000","last_peered":"2024-03-16T20:34:51.412944+0000","last_clean":"2024-03-16T20:34:51.412944+0000","last_became_active":"2024-03-16T20:33:53.881030+0000","last_became_peered":"2024-03-16T20:33:53.881030+0000","last_unstale":"2024-03-16T20:34:51.412944+0000","last_undegraded":"2024-03-16T20:34:51.412944+0000","last_fullsized":"2024-03-16T20:34:51.412944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:46.499466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417928+0000","last_change":"2024-03-16T20:33:48.853138+0000","last_active":"2024-03-16T20:34:51.417928+0000","last_peered":"2024-03-16T20:34:51.417928+0000","last_clean":"2024-03-16T20:34:51.417928+0000","last_became_active":"2024-03-16T20:33:48.851501+0000","last_became_peered":"2024-03-16T20:33:48.851501+0000","last_unstale":"2024-03-16T20:34:51.417928+0000","last_undegraded":"2024-03-16T20:34:51.417928+0000","last_fullsized":"2024-03-16T20:34:51.417928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:57:40.368614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414472+0000","last_change":"2024-03-16T20:33:49.850307+0000","last_active":"2024-03-16T20:34:51.414472+0000","last_peered":"2024-03-16T20:34:51.414472+0000","last_clean":"2024-03-16T20:34:51.414472+0000","last_became_active":"2024-03-16T20:33:49.849953+0000","last_became_peered":"2024-03-16T20:33:49.849953+0000","last_unstale":"2024-03-16T20:34:51.414472+0000","last_undegraded":"2024-03-16T20:34:51.414472+0000","last_fullsized":"2024-03-16T20:34:51.414472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:36:51.020011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415217+0000","last_change":"2024-03-16T20:33:53.892370+0000","last_active":"2024-03-16T20:34:51.415217+0000","last_peered":"2024-03-16T20:34:51.415217+0000","last_clean":"2024-03-16T20:34:51.415217+0000","last_became_active":"2024-03-16T20:33:53.891885+0000","last_became_peered":"2024-03-16T20:33:53.891885+0000","last_unstale":"2024-03-16T20:34:51.415217+0000","last_undegraded":"2024-03-16T20:34:51.415217+0000","last_fullsized":"2024-03-16T20:34:51.415217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:30.770882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.425434+0000","last_change":"2024-03-16T20:34:51.424981+0000","last_active":"2024-03-16T20:34:51.425434+0000","last_peered":"2024-03-16T20:34:51.425434+0000","last_clean":"2024-03-16T20:34:51.425434+0000","last_became_active":"2024-03-16T20:34:51.420101+0000","last_became_peered":"2024-03-16T20:34:51.420101+0000","last_unstale":"2024-03-16T20:34:51.425434+0000","last_undegraded":"2024-03-16T20:34:51.425434+0000","last_fullsized":"2024-03-16T20:34:51.425434+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:32.567770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.980271+0000","last_change":"2024-03-16T20:33:49.861637+0000","last_active":"2024-03-16T20:35:19.980271+0000","last_peered":"2024-03-16T20:35:19.980271+0000","last_clean":"2024-03-16T20:35:19.980271+0000","last_became_active":"2024-03-16T20:33:49.861284+0000","last_became_peered":"2024-03-16T20:33:49.861284+0000","last_unstale":"2024-03-16T20:35:19.980271+0000","last_undegraded":"2024-03-16T20:35:19.980271+0000","last_fullsized":"2024-03-16T20:35:19.980271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.937928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415270+0000","last_change":"2024-03-16T20:33:53.931633+0000","last_active":"2024-03-16T20:34:51.415270+0000","last_peered":"2024-03-16T20:34:51.415270+0000","last_clean":"2024-03-16T20:34:51.415270+0000","last_became_active":"2024-03-16T20:33:53.930633+0000","last_became_peered":"2024-03-16T20:33:53.930633+0000","last_unstale":"2024-03-16T20:34:51.415270+0000","last_undegraded":"2024-03-16T20:34:51.415270+0000","last_fullsized":"2024-03-16T20:34:51.415270+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:33:57.356317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.429155+0000","last_change":"2024-03-16T20:34:51.428773+0000","last_active":"2024-03-16T20:34:51.429155+0000","last_peered":"2024-03-16T20:34:51.429155+0000","last_clean":"2024-03-16T20:34:51.429155+0000","last_became_active":"2024-03-16T20:34:51.420784+0000","last_became_peered":"2024-03-16T20:34:51.420784+0000","last_unstale":"2024-03-16T20:34:51.429155+0000","last_undegraded":"2024-03-16T20:34:51.429155+0000","last_fullsized":"2024-03-16T20:34:51.429155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:39:54.702810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418483+0000","last_change":"2024-03-16T20:33:49.859085+0000","last_active":"2024-03-16T20:34:51.418483+0000","last_peered":"2024-03-16T20:34:51.418483+0000","last_clean":"2024-03-16T20:34:51.418483+0000","last_became_active":"2024-03-16T20:33:49.858637+0000","last_became_peered":"2024-03-16T20:33:49.858637+0000","last_unstale":"2024-03-16T20:34:51.418483+0000","last_undegraded":"2024-03-16T20:34:51.418483+0000","last_fullsized":"2024-03-16T20:34:51.418483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:22:17.020669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907388+0000","last_change":"2024-03-16T20:33:53.891866+0000","last_active":"2024-03-16T20:34:51.907388+0000","last_peered":"2024-03-16T20:34:51.907388+0000","last_clean":"2024-03-16T20:34:51.907388+0000","last_became_active":"2024-03-16T20:33:53.891424+0000","last_became_peered":"2024-03-16T20:33:53.891424+0000","last_unstale":"2024-03-16T20:34:51.907388+0000","last_undegraded":"2024-03-16T20:34:51.907388+0000","last_fullsized":"2024-03-16T20:34:51.907388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.036662+0000","last_change":"2024-03-16T20:33:48.856124+0000","last_active":"2024-03-16T20:35:19.036662+0000","last_peered":"2024-03-16T20:35:19.036662+0000","last_clean":"2024-03-16T20:35:19.036662+0000","last_became_active":"2024-03-16T20:33:48.855813+0000","last_became_peered":"2024-03-16T20:33:48.855813+0000","last_unstale":"2024-03-16T20:35:19.036662+0000","last_undegraded":"2024-03-16T20:35:19.036662+0000","last_fullsized":"2024-03-16T20:35:19.036662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:55.122457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419046+0000","last_change":"2024-03-16T20:33:49.857479+0000","last_active":"2024-03-16T20:34:51.419046+0000","last_peered":"2024-03-16T20:34:51.419046+0000","last_clean":"2024-03-16T20:34:51.419046+0000","last_became_active":"2024-03-16T20:33:49.856121+0000","last_became_peered":"2024-03-16T20:33:49.856121+0000","last_unstale":"2024-03-16T20:34:51.419046+0000","last_undegraded":"2024-03-16T20:34:51.419046+0000","last_fullsized":"2024-03-16T20:34:51.419046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:08:55.613977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.909071+0000","last_change":"2024-03-16T20:34:51.909071+0000","last_active":"2024-03-16T20:34:51.909071+0000","last_peered":"2024-03-16T20:34:51.909071+0000","last_clean":"2024-03-16T20:34:51.909071+0000","last_became_active":"2024-03-16T20:34:51.908570+0000","last_became_peered":"2024-03-16T20:34:51.908570+0000","last_unstale":"2024-03-16T20:34:51.909071+0000","last_undegraded":"2024-03-16T20:34:51.909071+0000","last_fullsized":"2024-03-16T20:34:51.909071+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:18.023401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907708+0000","last_change":"2024-03-16T20:33:48.846628+0000","last_active":"2024-03-16T20:34:51.907708+0000","last_peered":"2024-03-16T20:34:51.907708+0000","last_clean":"2024-03-16T20:34:51.907708+0000","last_became_active":"2024-03-16T20:33:48.846276+0000","last_became_peered":"2024-03-16T20:33:48.846276+0000","last_unstale":"2024-03-16T20:34:51.907708+0000","last_undegraded":"2024-03-16T20:34:51.907708+0000","last_fullsized":"2024-03-16T20:34:51.907708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:24.323938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418252+0000","last_change":"2024-03-16T20:33:49.854041+0000","last_active":"2024-03-16T20:34:51.418252+0000","last_peered":"2024-03-16T20:34:51.418252+0000","last_clean":"2024-03-16T20:34:51.418252+0000","last_became_active":"2024-03-16T20:33:49.853653+0000","last_became_peered":"2024-03-16T20:33:49.853653+0000","last_unstale":"2024-03-16T20:34:51.418252+0000","last_undegraded":"2024-03-16T20:34:51.418252+0000","last_fullsized":"2024-03-16T20:34:51.418252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:59:10.498314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.906967+0000","last_change":"2024-03-16T20:33:53.884292+0000","last_active":"2024-03-16T20:34:51.906967+0000","last_peered":"2024-03-16T20:34:51.906967+0000","last_clean":"2024-03-16T20:34:51.906967+0000","last_became_active":"2024-03-16T20:33:53.883778+0000","last_became_peered":"2024-03-16T20:33:53.883778+0000","last_unstale":"2024-03-16T20:34:51.906967+0000","last_undegraded":"2024-03-16T20:34:51.906967+0000","last_fullsized":"2024-03-16T20:34:51.906967+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:33.877142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416450+0000","last_change":"2024-03-16T20:33:48.845277+0000","last_active":"2024-03-16T20:34:51.416450+0000","last_peered":"2024-03-16T20:34:51.416450+0000","last_clean":"2024-03-16T20:34:51.416450+0000","last_became_active":"2024-03-16T20:33:48.844954+0000","last_became_peered":"2024-03-16T20:33:48.844954+0000","last_unstale":"2024-03-16T20:34:51.416450+0000","last_undegraded":"2024-03-16T20:34:51.416450+0000","last_fullsized":"2024-03-16T20:34:51.416450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:24:59.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418737+0000","last_change":"2024-03-16T20:33:49.856152+0000","last_active":"2024-03-16T20:34:51.418737+0000","last_peered":"2024-03-16T20:34:51.418737+0000","last_clean":"2024-03-16T20:34:51.418737+0000","last_became_active":"2024-03-16T20:33:49.855026+0000","last_became_peered":"2024-03-16T20:33:49.855026+0000","last_unstale":"2024-03-16T20:34:51.418737+0000","last_undegraded":"2024-03-16T20:34:51.418737+0000","last_fullsized":"2024-03-16T20:34:51.418737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:40.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414576+0000","last_change":"2024-03-16T20:33:49.851539+0000","last_active":"2024-03-16T20:34:51.414576+0000","last_peered":"2024-03-16T20:34:51.414576+0000","last_clean":"2024-03-16T20:34:51.414576+0000","last_became_active":"2024-03-16T20:33:49.851188+0000","last_became_peered":"2024-03-16T20:33:49.851188+0000","last_unstale":"2024-03-16T20:34:51.414576+0000","last_undegraded":"2024-03-16T20:34:51.414576+0000","last_fullsized":"2024-03-16T20:34:51.414576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:34:00.153832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976313+0000","last_change":"2024-03-16T20:33:49.845150+0000","last_active":"2024-03-16T20:35:19.976313+0000","last_peered":"2024-03-16T20:35:19.976313+0000","last_clean":"2024-03-16T20:35:19.976313+0000","last_became_active":"2024-03-16T20:33:49.844531+0000","last_became_peered":"2024-03-16T20:33:49.844531+0000","last_unstale":"2024-03-16T20:35:19.976313+0000","last_undegraded":"2024-03-16T20:35:19.976313+0000","last_fullsized":"2024-03-16T20:35:19.976313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:54.607559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.970119+0000","last_change":"2024-03-16T20:33:49.853608+0000","last_active":"2024-03-16T20:35:19.970119+0000","last_peered":"2024-03-16T20:35:19.970119+0000","last_clean":"2024-03-16T20:35:19.970119+0000","last_became_active":"2024-03-16T20:33:49.852800+0000","last_became_peered":"2024-03-16T20:33:49.852800+0000","last_unstale":"2024-03-16T20:35:19.970119+0000","last_undegraded":"2024-03-16T20:35:19.970119+0000","last_fullsized":"2024-03-16T20:35:19.970119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:59.988313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414491+0000","last_change":"2024-03-16T20:33:49.863542+0000","last_active":"2024-03-16T20:34:51.414491+0000","last_peered":"2024-03-16T20:34:51.414491+0000","last_clean":"2024-03-16T20:34:51.414491+0000","last_became_active":"2024-03-16T20:33:49.863143+0000","last_became_peered":"2024-03-16T20:33:49.863143+0000","last_unstale":"2024-03-16T20:34:51.414491+0000","last_undegraded":"2024-03-16T20:34:51.414491+0000","last_fullsized":"2024-03-16T20:34:51.414491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:18:29.997097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.984190+0000","last_change":"2024-03-16T20:34:51.418944+0000","last_active":"2024-03-16T20:35:19.984190+0000","last_peered":"2024-03-16T20:35:19.984190+0000","last_clean":"2024-03-16T20:35:19.984190+0000","last_became_active":"2024-03-16T20:34:51.418559+0000","last_became_peered":"2024-03-16T20:34:51.418559+0000","last_unstale":"2024-03-16T20:35:19.984190+0000","last_undegraded":"2024-03-16T20:35:19.984190+0000","last_fullsized":"2024-03-16T20:35:19.984190+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:59:27.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415009+0000","last_change":"2024-03-16T20:33:49.853004+0000","last_active":"2024-03-16T20:34:51.415009+0000","last_peered":"2024-03-16T20:34:51.415009+0000","last_clean":"2024-03-16T20:34:51.415009+0000","last_became_active":"2024-03-16T20:33:49.852609+0000","last_became_peered":"2024-03-16T20:33:49.852609+0000","last_unstale":"2024-03-16T20:34:51.415009+0000","last_undegraded":"2024-03-16T20:34:51.415009+0000","last_fullsized":"2024-03-16T20:34:51.415009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:42:58.648008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.989803+0000","last_change":"2024-03-16T20:33:49.857259+0000","last_active":"2024-03-16T20:35:19.989803+0000","last_peered":"2024-03-16T20:35:19.989803+0000","last_clean":"2024-03-16T20:35:19.989803+0000","last_became_active":"2024-03-16T20:33:49.856898+0000","last_became_peered":"2024-03-16T20:33:49.856898+0000","last_unstale":"2024-03-16T20:35:19.989803+0000","last_undegraded":"2024-03-16T20:35:19.989803+0000","last_fullsized":"2024-03-16T20:35:19.989803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:54.203804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976493+0000","last_change":"2024-03-16T20:33:49.860995+0000","last_active":"2024-03-16T20:35:19.976493+0000","last_peered":"2024-03-16T20:35:19.976493+0000","last_clean":"2024-03-16T20:35:19.976493+0000","last_became_active":"2024-03-16T20:33:49.860635+0000","last_became_peered":"2024-03-16T20:33:49.860635+0000","last_unstale":"2024-03-16T20:35:19.976493+0000","last_undegraded":"2024-03-16T20:35:19.976493+0000","last_fullsized":"2024-03-16T20:35:19.976493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:28:59.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907346+0000","last_change":"2024-03-16T20:33:49.854841+0000","last_active":"2024-03-16T20:34:51.907346+0000","last_peered":"2024-03-16T20:34:51.907346+0000","last_clean":"2024-03-16T20:34:51.907346+0000","last_became_active":"2024-03-16T20:33:49.854468+0000","last_became_peered":"2024-03-16T20:33:49.854468+0000","last_unstale":"2024-03-16T20:34:51.907346+0000","last_undegraded":"2024-03-16T20:34:51.907346+0000","last_fullsized":"2024-03-16T20:34:51.907346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:52.224203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979399+0000","last_change":"2024-03-16T20:33:49.854757+0000","last_active":"2024-03-16T20:35:19.979399+0000","last_peered":"2024-03-16T20:35:19.979399+0000","last_clean":"2024-03-16T20:35:19.979399+0000","last_became_active":"2024-03-16T20:33:49.852131+0000","last_became_peered":"2024-03-16T20:33:49.852131+0000","last_unstale":"2024-03-16T20:35:19.979399+0000","last_undegraded":"2024-03-16T20:35:19.979399+0000","last_fullsized":"2024-03-16T20:35:19.979399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:14.731341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994849+0000","last_change":"2024-03-16T20:33:49.863332+0000","last_active":"2024-03-16T20:35:19.994849+0000","last_peered":"2024-03-16T20:35:19.994849+0000","last_clean":"2024-03-16T20:35:19.994849+0000","last_became_active":"2024-03-16T20:33:49.862953+0000","last_became_peered":"2024-03-16T20:33:49.862953+0000","last_unstale":"2024-03-16T20:35:19.994849+0000","last_undegraded":"2024-03-16T20:35:19.994849+0000","last_fullsized":"2024-03-16T20:35:19.994849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:17.126504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983077+0000","last_change":"2024-03-16T20:33:49.849055+0000","last_active":"2024-03-16T20:35:19.983077+0000","last_peered":"2024-03-16T20:35:19.983077+0000","last_clean":"2024-03-16T20:35:19.983077+0000","last_became_active":"2024-03-16T20:33:49.848686+0000","last_became_peered":"2024-03-16T20:33:49.848686+0000","last_unstale":"2024-03-16T20:35:19.983077+0000","last_undegraded":"2024-03-16T20:35:19.983077+0000","last_fullsized":"2024-03-16T20:35:19.983077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:15.783993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418207+0000","last_change":"2024-03-16T20:34:51.418207+0000","last_active":"2024-03-16T20:34:51.418207+0000","last_peered":"2024-03-16T20:34:51.418207+0000","last_clean":"2024-03-16T20:34:51.418207+0000","last_became_active":"2024-03-16T20:34:51.417875+0000","last_became_peered":"2024-03-16T20:34:51.417875+0000","last_unstale":"2024-03-16T20:34:51.418207+0000","last_undegraded":"2024-03-16T20:34:51.418207+0000","last_fullsized":"2024-03-16T20:34:51.418207+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:46:11.734927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":291,"ondisk_log_size":291,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286596,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68284,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":752,"num_read_kb":454,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":204993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":507120,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1155407,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":511146,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":507428,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1103925,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":502554,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":565055,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":504010,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":565417,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":502262,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":506628,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1099507,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:35:23.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T20:35:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:24.059 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:23 smithi023 ceph-mon[31879]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:24.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:23 smithi031 ceph-mon[36281]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:24.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:23 smithi189 ceph-mon[36655]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:24.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:24 smithi023 ceph-mon[31879]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:35:25.188 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:24 smithi031 ceph-mon[36281]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:35:25.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:24 smithi189 ceph-mon[36655]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-16T20:35:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 -- 172.21.15.23:0/2041147001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c101880 msgr2=0x7f1f8c10dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 --2- 172.21.15.23:0/2041147001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c101880 0x7f1f8c10dd60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1f8400f530 tx=0x7f1f8400f840 comp rx=0 tx=0).stop 2024-03-16T20:35:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 -- 172.21.15.23:0/2041147001 shutdown_connections 2024-03-16T20:35:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 --2- 172.21.15.23:0/2041147001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c101880 0x7f1f8c10dd60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 --2- 172.21.15.23:0/2041147001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 --2- 172.21.15.23:0/2041147001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 -- 172.21.15.23:0/2041147001 >> 172.21.15.23:0/2041147001 conn(0x7f1f8c077e10 msgr2=0x7f1f8c078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 -- 172.21.15.23:0/2041147001 shutdown_connections 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.451+0000 7f1f920a4700 1 -- 172.21.15.23:0/2041147001 wait complete. 2024-03-16T20:35:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 Processor -- start 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 -- start start 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 0x7f1f8c196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c19b2e0 con 0x7f1f8c100ee0 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c19b450 con 0x7f1f8c106f20 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f920a4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c19b610 con 0x7f1f8c101880 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47912/0 (socket says 172.21.15.23:47912) 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f8b7fe700 1 -- 172.21.15.23:0/56634012 learned_addr learned my addr 172.21.15.23:0/56634012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f8affd700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 0x7f1f8c196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.452+0000 7f1f8bfff700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 -- 172.21.15.23:0/56634012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 msgr2=0x7f1f8c196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 0x7f1f8c196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 -- 172.21.15.23:0/56634012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 msgr2=0x7f1f8c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 -- 172.21.15.23:0/56634012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009d00 con 0x7f1f8c100ee0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8affd700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 0x7f1f8c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8bfff700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f8b7fe700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00e9a0 tx=0x7f1f7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c009790 con 0x7f1f8c100ee0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c19b8f0 con 0x7f1f8c100ee0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f7c010430 con 0x7f1f8c100ee0 2024-03-16T20:35:25.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c018680 con 0x7f1f8c100ee0 2024-03-16T20:35:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.453+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c1a3150 con 0x7f1f8c100ee0 2024-03-16T20:35:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.454+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c109320 con 0x7f1f8c100ee0 2024-03-16T20:35:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.455+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1f7c010910 con 0x7f1f8c100ee0 2024-03-16T20:35:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.456+0000 7f1f88ff9700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 0x7f1f740a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.456+0000 7f1f8affd700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 0x7f1f740a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.456+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f1f7c014070 con 0x7f1f8c100ee0 2024-03-16T20:35:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.456+0000 7f1f8affd700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 0x7f1f740a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c078550 tx=0x7f1f8000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:25.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.460+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1f7c096bf0 con 0x7f1f8c100ee0 2024-03-16T20:35:25.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.656+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f8c1977c0 con 0x7f1f740a1ac0 2024-03-16T20:35:25.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.666+0000 7f1f88ff9700 1 -- 172.21.15.23:0/56634012 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340889 (secure 0 0 0) 0x7f1f8c1977c0 con 0x7f1f740a1ac0 2024-03-16T20:35:25.674 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 msgr2=0x7f1f740a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 0x7f1f740a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c078550 tx=0x7f1f8000d040 comp rx=0 tx=0).stop 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 msgr2=0x7f1f8c196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00e9a0 tx=0x7f1f7c00ecb0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 shutdown_connections 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1f740a1ac0 0x7f1f740a3f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f8c100ee0 0x7f1f8c196400 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f8c106f20 0x7f1f8c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 --2- 172.21.15.23:0/56634012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8c101880 0x7f1f8c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:25.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 >> 172.21.15.23:0/56634012 conn(0x7f1f8c077e10 msgr2=0x7f1f8c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:25.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 shutdown_connections 2024-03-16T20:35:25.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:25.669+0000 7f1f920a4700 1 -- 172.21.15.23:0/56634012 wait complete. 2024-03-16T20:35:25.678 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-16T20:35:26.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:26 smithi023 ceph-mon[31879]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:26.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:26 smithi023 ceph-mon[31879]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:26.434 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-16T20:35:23.965125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658886,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":878,"num_read_kb":563,"num_write":427,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378264,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563560,"statfs":{"total":1151940427776,"available":1151553085440,"internally_reserved":0,"allocated":12738560,"data_stored":8030459,"data_compressed":357126,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":101,"num_read_kb":51,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.987879+0000","last_change":"2024-03-16T20:33:49.858111+0000","last_active":"2024-03-16T20:35:19.987879+0000","last_peered":"2024-03-16T20:35:19.987879+0000","last_clean":"2024-03-16T20:35:19.987879+0000","last_became_active":"2024-03-16T20:33:49.857656+0000","last_became_peered":"2024-03-16T20:33:49.857656+0000","last_unstale":"2024-03-16T20:35:19.987879+0000","last_undegraded":"2024-03-16T20:35:19.987879+0000","last_fullsized":"2024-03-16T20:35:19.987879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:16:47.779595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.978153+0000","last_change":"2024-03-16T20:33:49.854850+0000","last_active":"2024-03-16T20:35:19.978153+0000","last_peered":"2024-03-16T20:35:19.978153+0000","last_clean":"2024-03-16T20:35:19.978153+0000","last_became_active":"2024-03-16T20:33:49.854399+0000","last_became_peered":"2024-03-16T20:33:49.854399+0000","last_unstale":"2024-03-16T20:35:19.978153+0000","last_undegraded":"2024-03-16T20:35:19.978153+0000","last_fullsized":"2024-03-16T20:35:19.978153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:55.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.990202+0000","last_change":"2024-03-16T20:33:49.858831+0000","last_active":"2024-03-16T20:35:19.990202+0000","last_peered":"2024-03-16T20:35:19.990202+0000","last_clean":"2024-03-16T20:35:19.990202+0000","last_became_active":"2024-03-16T20:33:49.858507+0000","last_became_peered":"2024-03-16T20:33:49.858507+0000","last_unstale":"2024-03-16T20:35:19.990202+0000","last_undegraded":"2024-03-16T20:35:19.990202+0000","last_fullsized":"2024-03-16T20:35:19.990202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:01:35.447285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979087+0000","last_change":"2024-03-16T20:33:49.852269+0000","last_active":"2024-03-16T20:35:19.979087+0000","last_peered":"2024-03-16T20:35:19.979087+0000","last_clean":"2024-03-16T20:35:19.979087+0000","last_became_active":"2024-03-16T20:33:49.851904+0000","last_became_peered":"2024-03-16T20:33:49.851904+0000","last_unstale":"2024-03-16T20:35:19.979087+0000","last_undegraded":"2024-03-16T20:35:19.979087+0000","last_fullsized":"2024-03-16T20:35:19.979087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:38.520111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.961191+0000","last_change":"2024-03-16T20:33:49.852371+0000","last_active":"2024-03-16T20:35:19.961191+0000","last_peered":"2024-03-16T20:35:19.961191+0000","last_clean":"2024-03-16T20:35:19.961191+0000","last_became_active":"2024-03-16T20:33:49.852034+0000","last_became_peered":"2024-03-16T20:33:49.852034+0000","last_unstale":"2024-03-16T20:35:19.961191+0000","last_undegraded":"2024-03-16T20:35:19.961191+0000","last_fullsized":"2024-03-16T20:35:19.961191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:40:10.229852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.971913+0000","last_change":"2024-03-16T20:33:49.855209+0000","last_active":"2024-03-16T20:35:19.971913+0000","last_peered":"2024-03-16T20:35:19.971913+0000","last_clean":"2024-03-16T20:35:19.971913+0000","last_became_active":"2024-03-16T20:33:49.854502+0000","last_became_peered":"2024-03-16T20:33:49.854502+0000","last_unstale":"2024-03-16T20:35:19.971913+0000","last_undegraded":"2024-03-16T20:35:19.971913+0000","last_fullsized":"2024-03-16T20:35:19.971913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:47.792663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417926+0000","last_change":"2024-03-16T20:33:49.852668+0000","last_active":"2024-03-16T20:34:51.417926+0000","last_peered":"2024-03-16T20:34:51.417926+0000","last_clean":"2024-03-16T20:34:51.417926+0000","last_became_active":"2024-03-16T20:33:49.852263+0000","last_became_peered":"2024-03-16T20:33:49.852263+0000","last_unstale":"2024-03-16T20:34:51.417926+0000","last_undegraded":"2024-03-16T20:34:51.417926+0000","last_fullsized":"2024-03-16T20:34:51.417926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:49.815894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.981712+0000","last_change":"2024-03-16T20:33:49.854962+0000","last_active":"2024-03-16T20:35:19.981712+0000","last_peered":"2024-03-16T20:35:19.981712+0000","last_clean":"2024-03-16T20:35:19.981712+0000","last_became_active":"2024-03-16T20:33:49.853963+0000","last_became_peered":"2024-03-16T20:33:49.853963+0000","last_unstale":"2024-03-16T20:35:19.981712+0000","last_undegraded":"2024-03-16T20:35:19.981712+0000","last_fullsized":"2024-03-16T20:35:19.981712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:37:19.213607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.992384+0000","last_change":"2024-03-16T20:33:49.857741+0000","last_active":"2024-03-16T20:35:19.992384+0000","last_peered":"2024-03-16T20:35:19.992384+0000","last_clean":"2024-03-16T20:35:19.992384+0000","last_became_active":"2024-03-16T20:33:49.857354+0000","last_became_peered":"2024-03-16T20:33:49.857354+0000","last_unstale":"2024-03-16T20:35:19.992384+0000","last_undegraded":"2024-03-16T20:35:19.992384+0000","last_fullsized":"2024-03-16T20:35:19.992384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:04:22.085460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.982596+0000","last_change":"2024-03-16T20:33:49.864008+0000","last_active":"2024-03-16T20:35:19.982596+0000","last_peered":"2024-03-16T20:35:19.982596+0000","last_clean":"2024-03-16T20:35:19.982596+0000","last_became_active":"2024-03-16T20:33:49.862779+0000","last_became_peered":"2024-03-16T20:33:49.862779+0000","last_unstale":"2024-03-16T20:35:19.982596+0000","last_undegraded":"2024-03-16T20:35:19.982596+0000","last_fullsized":"2024-03-16T20:35:19.982596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:34:23.249973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419001+0000","last_change":"2024-03-16T20:33:49.858951+0000","last_active":"2024-03-16T20:34:51.419001+0000","last_peered":"2024-03-16T20:34:51.419001+0000","last_clean":"2024-03-16T20:34:51.419001+0000","last_became_active":"2024-03-16T20:33:49.858575+0000","last_became_peered":"2024-03-16T20:33:49.858575+0000","last_unstale":"2024-03-16T20:34:51.419001+0000","last_undegraded":"2024-03-16T20:34:51.419001+0000","last_fullsized":"2024-03-16T20:34:51.419001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:07:01.030826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.960235+0000","last_change":"2024-03-16T20:33:49.854399+0000","last_active":"2024-03-16T20:35:19.960235+0000","last_peered":"2024-03-16T20:35:19.960235+0000","last_clean":"2024-03-16T20:35:19.960235+0000","last_became_active":"2024-03-16T20:33:49.853937+0000","last_became_peered":"2024-03-16T20:33:49.853937+0000","last_unstale":"2024-03-16T20:35:19.960235+0000","last_undegraded":"2024-03-16T20:35:19.960235+0000","last_fullsized":"2024-03-16T20:35:19.960235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:48:00.207213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983196+0000","last_change":"2024-03-16T20:33:49.853101+0000","last_active":"2024-03-16T20:35:19.983196+0000","last_peered":"2024-03-16T20:35:19.983196+0000","last_clean":"2024-03-16T20:35:19.983196+0000","last_became_active":"2024-03-16T20:33:49.852667+0000","last_became_peered":"2024-03-16T20:33:49.852667+0000","last_unstale":"2024-03-16T20:35:19.983196+0000","last_undegraded":"2024-03-16T20:35:19.983196+0000","last_fullsized":"2024-03-16T20:35:19.983196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:17:43.047348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.977844+0000","last_change":"2024-03-16T20:33:49.852907+0000","last_active":"2024-03-16T20:35:19.977844+0000","last_peered":"2024-03-16T20:35:19.977844+0000","last_clean":"2024-03-16T20:35:19.977844+0000","last_became_active":"2024-03-16T20:33:49.852552+0000","last_became_peered":"2024-03-16T20:33:49.852552+0000","last_unstale":"2024-03-16T20:35:19.977844+0000","last_undegraded":"2024-03-16T20:35:19.977844+0000","last_fullsized":"2024-03-16T20:35:19.977844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:46:34.019736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.971987+0000","last_change":"2024-03-16T20:33:49.863477+0000","last_active":"2024-03-16T20:35:19.971987+0000","last_peered":"2024-03-16T20:35:19.971987+0000","last_clean":"2024-03-16T20:35:19.971987+0000","last_became_active":"2024-03-16T20:33:49.863163+0000","last_became_peered":"2024-03-16T20:33:49.863163+0000","last_unstale":"2024-03-16T20:35:19.971987+0000","last_undegraded":"2024-03-16T20:35:19.971987+0000","last_fullsized":"2024-03-16T20:35:19.971987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:46:55.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994338+0000","last_change":"2024-03-16T20:33:49.863312+0000","last_active":"2024-03-16T20:35:19.994338+0000","last_peered":"2024-03-16T20:35:19.994338+0000","last_clean":"2024-03-16T20:35:19.994338+0000","last_became_active":"2024-03-16T20:33:49.862918+0000","last_became_peered":"2024-03-16T20:33:49.862918+0000","last_unstale":"2024-03-16T20:35:19.994338+0000","last_undegraded":"2024-03-16T20:35:19.994338+0000","last_fullsized":"2024-03-16T20:35:19.994338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:24:33.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.989717+0000","last_change":"2024-03-16T20:33:49.858863+0000","last_active":"2024-03-16T20:35:19.989717+0000","last_peered":"2024-03-16T20:35:19.989717+0000","last_clean":"2024-03-16T20:35:19.989717+0000","last_became_active":"2024-03-16T20:33:49.858543+0000","last_became_peered":"2024-03-16T20:33:49.858543+0000","last_unstale":"2024-03-16T20:35:19.989717+0000","last_undegraded":"2024-03-16T20:35:19.989717+0000","last_fullsized":"2024-03-16T20:35:19.989717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:04.584473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976378+0000","last_change":"2024-03-16T20:33:49.857863+0000","last_active":"2024-03-16T20:35:19.976378+0000","last_peered":"2024-03-16T20:35:19.976378+0000","last_clean":"2024-03-16T20:35:19.976378+0000","last_became_active":"2024-03-16T20:33:49.857503+0000","last_became_peered":"2024-03-16T20:33:49.857503+0000","last_unstale":"2024-03-16T20:35:19.976378+0000","last_undegraded":"2024-03-16T20:35:19.976378+0000","last_fullsized":"2024-03-16T20:35:19.976378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:06:11.206926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.977578+0000","last_change":"2024-03-16T20:33:49.849940+0000","last_active":"2024-03-16T20:35:19.977578+0000","last_peered":"2024-03-16T20:35:19.977578+0000","last_clean":"2024-03-16T20:35:19.977578+0000","last_became_active":"2024-03-16T20:33:49.849632+0000","last_became_peered":"2024-03-16T20:33:49.849632+0000","last_unstale":"2024-03-16T20:35:19.977578+0000","last_undegraded":"2024-03-16T20:35:19.977578+0000","last_fullsized":"2024-03-16T20:35:19.977578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:20:25.921270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418880+0000","last_change":"2024-03-16T20:33:53.882456+0000","last_active":"2024-03-16T20:34:51.418880+0000","last_peered":"2024-03-16T20:34:51.418880+0000","last_clean":"2024-03-16T20:34:51.418880+0000","last_became_active":"2024-03-16T20:33:53.882040+0000","last_became_peered":"2024-03-16T20:33:53.882040+0000","last_unstale":"2024-03-16T20:34:51.418880+0000","last_undegraded":"2024-03-16T20:34:51.418880+0000","last_fullsized":"2024-03-16T20:34:51.418880+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:18:59.392600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":112,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:20.037290+0000","last_change":"2024-03-16T20:33:48.846129+0000","last_active":"2024-03-16T20:35:20.037290+0000","last_peered":"2024-03-16T20:35:20.037290+0000","last_clean":"2024-03-16T20:35:20.037290+0000","last_became_active":"2024-03-16T20:33:48.845783+0000","last_became_peered":"2024-03-16T20:33:48.845783+0000","last_unstale":"2024-03-16T20:35:20.037290+0000","last_undegraded":"2024-03-16T20:35:20.037290+0000","last_fullsized":"2024-03-16T20:35:20.037290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:46:03.456669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.973691+0000","last_change":"2024-03-16T20:33:49.851397+0000","last_active":"2024-03-16T20:35:19.973691+0000","last_peered":"2024-03-16T20:35:19.973691+0000","last_clean":"2024-03-16T20:35:19.973691+0000","last_became_active":"2024-03-16T20:33:49.851097+0000","last_became_peered":"2024-03-16T20:33:49.851097+0000","last_unstale":"2024-03-16T20:35:19.973691+0000","last_undegraded":"2024-03-16T20:35:19.973691+0000","last_fullsized":"2024-03-16T20:35:19.973691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:57:55.165017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414413+0000","last_change":"2024-03-16T20:33:53.893593+0000","last_active":"2024-03-16T20:34:51.414413+0000","last_peered":"2024-03-16T20:34:51.414413+0000","last_clean":"2024-03-16T20:34:51.414413+0000","last_became_active":"2024-03-16T20:33:53.893101+0000","last_became_peered":"2024-03-16T20:33:53.893101+0000","last_unstale":"2024-03-16T20:34:51.414413+0000","last_undegraded":"2024-03-16T20:34:51.414413+0000","last_fullsized":"2024-03-16T20:34:51.414413+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:22:59.472049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418047+0000","last_change":"2024-03-16T20:33:48.852786+0000","last_active":"2024-03-16T20:34:51.418047+0000","last_peered":"2024-03-16T20:34:51.418047+0000","last_clean":"2024-03-16T20:34:51.418047+0000","last_became_active":"2024-03-16T20:33:48.852477+0000","last_became_peered":"2024-03-16T20:33:48.852477+0000","last_unstale":"2024-03-16T20:34:51.418047+0000","last_undegraded":"2024-03-16T20:34:51.418047+0000","last_fullsized":"2024-03-16T20:34:51.418047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:20:49.191312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.990169+0000","last_change":"2024-03-16T20:33:49.859123+0000","last_active":"2024-03-16T20:35:19.990169+0000","last_peered":"2024-03-16T20:35:19.990169+0000","last_clean":"2024-03-16T20:35:19.990169+0000","last_became_active":"2024-03-16T20:33:49.858746+0000","last_became_peered":"2024-03-16T20:33:49.858746+0000","last_unstale":"2024-03-16T20:35:19.990169+0000","last_undegraded":"2024-03-16T20:35:19.990169+0000","last_fullsized":"2024-03-16T20:35:19.990169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:10:35.444073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.910052+0000","last_change":"2024-03-16T20:34:51.910052+0000","last_active":"2024-03-16T20:34:51.910052+0000","last_peered":"2024-03-16T20:34:51.910052+0000","last_clean":"2024-03-16T20:34:51.910052+0000","last_became_active":"2024-03-16T20:34:51.909334+0000","last_became_peered":"2024-03-16T20:34:51.909334+0000","last_unstale":"2024-03-16T20:34:51.910052+0000","last_undegraded":"2024-03-16T20:34:51.910052+0000","last_fullsized":"2024-03-16T20:34:51.910052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:49:08.179435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413662+0000","last_change":"2024-03-16T20:33:48.845405+0000","last_active":"2024-03-16T20:34:51.413662+0000","last_peered":"2024-03-16T20:34:51.413662+0000","last_clean":"2024-03-16T20:34:51.413662+0000","last_became_active":"2024-03-16T20:33:48.845033+0000","last_became_peered":"2024-03-16T20:33:48.845033+0000","last_unstale":"2024-03-16T20:34:51.413662+0000","last_undegraded":"2024-03-16T20:34:51.413662+0000","last_fullsized":"2024-03-16T20:34:51.413662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:52.632960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994318+0000","last_change":"2024-03-16T20:33:49.864934+0000","last_active":"2024-03-16T20:35:19.994318+0000","last_peered":"2024-03-16T20:35:19.994318+0000","last_clean":"2024-03-16T20:35:19.994318+0000","last_became_active":"2024-03-16T20:33:49.864574+0000","last_became_peered":"2024-03-16T20:33:49.864574+0000","last_unstale":"2024-03-16T20:35:19.994318+0000","last_undegraded":"2024-03-16T20:35:19.994318+0000","last_fullsized":"2024-03-16T20:35:19.994318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:20:07.010788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414410+0000","last_change":"2024-03-16T20:33:53.890446+0000","last_active":"2024-03-16T20:34:51.414410+0000","last_peered":"2024-03-16T20:34:51.414410+0000","last_clean":"2024-03-16T20:34:51.414410+0000","last_became_active":"2024-03-16T20:33:53.889832+0000","last_became_peered":"2024-03-16T20:33:53.889832+0000","last_unstale":"2024-03-16T20:34:51.414410+0000","last_undegraded":"2024-03-16T20:34:51.414410+0000","last_fullsized":"2024-03-16T20:34:51.414410+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:52:49.031999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413171+0000","last_change":"2024-03-16T20:33:48.845809+0000","last_active":"2024-03-16T20:34:51.413171+0000","last_peered":"2024-03-16T20:34:51.413171+0000","last_clean":"2024-03-16T20:34:51.413171+0000","last_became_active":"2024-03-16T20:33:48.844751+0000","last_became_peered":"2024-03-16T20:33:48.844751+0000","last_unstale":"2024-03-16T20:34:51.413171+0000","last_undegraded":"2024-03-16T20:34:51.413171+0000","last_fullsized":"2024-03-16T20:34:51.413171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:25:41.148435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907593+0000","last_change":"2024-03-16T20:33:49.857181+0000","last_active":"2024-03-16T20:34:51.907593+0000","last_peered":"2024-03-16T20:34:51.907593+0000","last_clean":"2024-03-16T20:34:51.907593+0000","last_became_active":"2024-03-16T20:33:49.856866+0000","last_became_peered":"2024-03-16T20:33:49.856866+0000","last_unstale":"2024-03-16T20:34:51.907593+0000","last_undegraded":"2024-03-16T20:34:51.907593+0000","last_fullsized":"2024-03-16T20:34:51.907593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.006531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419023+0000","last_change":"2024-03-16T20:33:53.879419+0000","last_active":"2024-03-16T20:34:51.419023+0000","last_peered":"2024-03-16T20:34:51.419023+0000","last_clean":"2024-03-16T20:34:51.419023+0000","last_became_active":"2024-03-16T20:33:53.878590+0000","last_became_peered":"2024-03-16T20:33:53.878590+0000","last_unstale":"2024-03-16T20:34:51.419023+0000","last_undegraded":"2024-03-16T20:34:51.419023+0000","last_fullsized":"2024-03-16T20:34:51.419023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:06:34.552834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:17.998840+0000","last_change":"2024-03-16T20:33:48.845122+0000","last_active":"2024-03-16T20:35:17.998840+0000","last_peered":"2024-03-16T20:35:17.998840+0000","last_clean":"2024-03-16T20:35:17.998840+0000","last_became_active":"2024-03-16T20:33:48.844109+0000","last_became_peered":"2024-03-16T20:33:48.844109+0000","last_unstale":"2024-03-16T20:35:17.998840+0000","last_undegraded":"2024-03-16T20:35:17.998840+0000","last_fullsized":"2024-03-16T20:35:17.998840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:05:46.911439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.957672+0000","last_change":"2024-03-16T20:33:49.850398+0000","last_active":"2024-03-16T20:35:19.957672+0000","last_peered":"2024-03-16T20:35:19.957672+0000","last_clean":"2024-03-16T20:35:19.957672+0000","last_became_active":"2024-03-16T20:33:49.850059+0000","last_became_peered":"2024-03-16T20:33:49.850059+0000","last_unstale":"2024-03-16T20:35:19.957672+0000","last_undegraded":"2024-03-16T20:35:19.957672+0000","last_fullsized":"2024-03-16T20:35:19.957672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:00:02.193759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419876+0000","last_change":"2024-03-16T20:33:53.890035+0000","last_active":"2024-03-16T20:34:51.419876+0000","last_peered":"2024-03-16T20:34:51.419876+0000","last_clean":"2024-03-16T20:34:51.419876+0000","last_became_active":"2024-03-16T20:33:53.889468+0000","last_became_peered":"2024-03-16T20:33:53.889468+0000","last_unstale":"2024-03-16T20:34:51.419876+0000","last_undegraded":"2024-03-16T20:34:51.419876+0000","last_fullsized":"2024-03-16T20:34:51.419876+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:11:47.125880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.760192+0000","last_change":"2024-03-16T20:33:48.850165+0000","last_active":"2024-03-16T20:34:51.760192+0000","last_peered":"2024-03-16T20:34:51.760192+0000","last_clean":"2024-03-16T20:34:51.760192+0000","last_became_active":"2024-03-16T20:33:48.849826+0000","last_became_peered":"2024-03-16T20:33:48.849826+0000","last_unstale":"2024-03-16T20:34:51.760192+0000","last_undegraded":"2024-03-16T20:34:51.760192+0000","last_fullsized":"2024-03-16T20:34:51.760192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:46:23.790227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907632+0000","last_change":"2024-03-16T20:33:49.857506+0000","last_active":"2024-03-16T20:34:51.907632+0000","last_peered":"2024-03-16T20:34:51.907632+0000","last_clean":"2024-03-16T20:34:51.907632+0000","last_became_active":"2024-03-16T20:33:49.856343+0000","last_became_peered":"2024-03-16T20:33:49.856343+0000","last_unstale":"2024-03-16T20:34:51.907632+0000","last_undegraded":"2024-03-16T20:34:51.907632+0000","last_fullsized":"2024-03-16T20:34:51.907632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:18:02.747518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.420360+0000","last_change":"2024-03-16T20:34:51.420360+0000","last_active":"2024-03-16T20:34:51.420360+0000","last_peered":"2024-03-16T20:34:51.420360+0000","last_clean":"2024-03-16T20:34:51.420360+0000","last_became_active":"2024-03-16T20:34:51.419906+0000","last_became_peered":"2024-03-16T20:34:51.419906+0000","last_unstale":"2024-03-16T20:34:51.420360+0000","last_undegraded":"2024-03-16T20:34:51.420360+0000","last_fullsized":"2024-03-16T20:34:51.420360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:47.903906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415964+0000","last_change":"2024-03-16T20:33:48.849773+0000","last_active":"2024-03-16T20:34:51.415964+0000","last_peered":"2024-03-16T20:34:51.415964+0000","last_clean":"2024-03-16T20:34:51.415964+0000","last_became_active":"2024-03-16T20:33:48.848736+0000","last_became_peered":"2024-03-16T20:33:48.848736+0000","last_unstale":"2024-03-16T20:34:51.415964+0000","last_undegraded":"2024-03-16T20:34:51.415964+0000","last_fullsized":"2024-03-16T20:34:51.415964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:54:37.017119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983450+0000","last_change":"2024-03-16T20:33:49.853446+0000","last_active":"2024-03-16T20:35:19.983450+0000","last_peered":"2024-03-16T20:35:19.983450+0000","last_clean":"2024-03-16T20:35:19.983450+0000","last_became_active":"2024-03-16T20:33:49.851886+0000","last_became_peered":"2024-03-16T20:33:49.851886+0000","last_unstale":"2024-03-16T20:35:19.983450+0000","last_undegraded":"2024-03-16T20:35:19.983450+0000","last_fullsized":"2024-03-16T20:35:19.983450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:43:39.675229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.414121+0000","last_change":"2024-03-16T20:33:53.890388+0000","last_active":"2024-03-16T20:34:51.414121+0000","last_peered":"2024-03-16T20:34:51.414121+0000","last_clean":"2024-03-16T20:34:51.414121+0000","last_became_active":"2024-03-16T20:33:53.889966+0000","last_became_peered":"2024-03-16T20:33:53.889966+0000","last_unstale":"2024-03-16T20:34:51.414121+0000","last_undegraded":"2024-03-16T20:34:51.414121+0000","last_fullsized":"2024-03-16T20:34:51.414121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:21.475392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907286+0000","last_change":"2024-03-16T20:33:48.841524+0000","last_active":"2024-03-16T20:34:51.907286+0000","last_peered":"2024-03-16T20:34:51.907286+0000","last_clean":"2024-03-16T20:34:51.907286+0000","last_became_active":"2024-03-16T20:33:48.841239+0000","last_became_peered":"2024-03-16T20:33:48.841239+0000","last_unstale":"2024-03-16T20:34:51.907286+0000","last_undegraded":"2024-03-16T20:34:51.907286+0000","last_fullsized":"2024-03-16T20:34:51.907286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:42.206113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979408+0000","last_change":"2024-03-16T20:33:49.856131+0000","last_active":"2024-03-16T20:35:19.979408+0000","last_peered":"2024-03-16T20:35:19.979408+0000","last_clean":"2024-03-16T20:35:19.979408+0000","last_became_active":"2024-03-16T20:33:49.855819+0000","last_became_peered":"2024-03-16T20:33:49.855819+0000","last_unstale":"2024-03-16T20:35:19.979408+0000","last_undegraded":"2024-03-16T20:35:19.979408+0000","last_fullsized":"2024-03-16T20:35:19.979408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:48:22.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417444+0000","last_change":"2024-03-16T20:33:53.888159+0000","last_active":"2024-03-16T20:34:51.417444+0000","last_peered":"2024-03-16T20:34:51.417444+0000","last_clean":"2024-03-16T20:34:51.417444+0000","last_became_active":"2024-03-16T20:33:53.887777+0000","last_became_peered":"2024-03-16T20:33:53.887777+0000","last_unstale":"2024-03-16T20:34:51.417444+0000","last_undegraded":"2024-03-16T20:34:51.417444+0000","last_fullsized":"2024-03-16T20:34:51.417444+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:26:17.021957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":176,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:21.001698+0000","last_change":"2024-03-16T20:33:48.846649+0000","last_active":"2024-03-16T20:35:21.001698+0000","last_peered":"2024-03-16T20:35:21.001698+0000","last_clean":"2024-03-16T20:35:21.001698+0000","last_became_active":"2024-03-16T20:33:48.846344+0000","last_became_peered":"2024-03-16T20:33:48.846344+0000","last_unstale":"2024-03-16T20:35:21.001698+0000","last_undegraded":"2024-03-16T20:35:21.001698+0000","last_fullsized":"2024-03-16T20:35:21.001698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:54:24.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994181+0000","last_change":"2024-03-16T20:33:49.866087+0000","last_active":"2024-03-16T20:35:19.994181+0000","last_peered":"2024-03-16T20:35:19.994181+0000","last_clean":"2024-03-16T20:35:19.994181+0000","last_became_active":"2024-03-16T20:33:49.865722+0000","last_became_peered":"2024-03-16T20:33:49.865722+0000","last_unstale":"2024-03-16T20:35:19.994181+0000","last_undegraded":"2024-03-16T20:35:19.994181+0000","last_fullsized":"2024-03-16T20:35:19.994181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:04:09.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412158+0000","last_change":"2024-03-16T20:33:53.890208+0000","last_active":"2024-03-16T20:34:51.412158+0000","last_peered":"2024-03-16T20:34:51.412158+0000","last_clean":"2024-03-16T20:34:51.412158+0000","last_became_active":"2024-03-16T20:33:53.889807+0000","last_became_peered":"2024-03-16T20:33:53.889807+0000","last_unstale":"2024-03-16T20:34:51.412158+0000","last_undegraded":"2024-03-16T20:34:51.412158+0000","last_fullsized":"2024-03-16T20:34:51.412158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:30:53.439687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418606+0000","last_change":"2024-03-16T20:33:48.851610+0000","last_active":"2024-03-16T20:34:51.418606+0000","last_peered":"2024-03-16T20:34:51.418606+0000","last_clean":"2024-03-16T20:34:51.418606+0000","last_became_active":"2024-03-16T20:33:48.851054+0000","last_became_peered":"2024-03-16T20:33:48.851054+0000","last_unstale":"2024-03-16T20:34:51.418606+0000","last_undegraded":"2024-03-16T20:34:51.418606+0000","last_fullsized":"2024-03-16T20:34:51.418606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:58:03.733318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.972423+0000","last_change":"2024-03-16T20:33:49.856534+0000","last_active":"2024-03-16T20:35:19.972423+0000","last_peered":"2024-03-16T20:35:19.972423+0000","last_clean":"2024-03-16T20:35:19.972423+0000","last_became_active":"2024-03-16T20:33:49.855957+0000","last_became_peered":"2024-03-16T20:33:49.855957+0000","last_unstale":"2024-03-16T20:35:19.972423+0000","last_undegraded":"2024-03-16T20:35:19.972423+0000","last_fullsized":"2024-03-16T20:35:19.972423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:51:07.035334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415269+0000","last_change":"2024-03-16T20:33:53.892149+0000","last_active":"2024-03-16T20:34:51.415269+0000","last_peered":"2024-03-16T20:34:51.415269+0000","last_clean":"2024-03-16T20:34:51.415269+0000","last_became_active":"2024-03-16T20:33:53.891730+0000","last_became_peered":"2024-03-16T20:33:53.891730+0000","last_unstale":"2024-03-16T20:34:51.415269+0000","last_undegraded":"2024-03-16T20:34:51.415269+0000","last_fullsized":"2024-03-16T20:34:51.415269+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:59.507741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907483+0000","last_change":"2024-03-16T20:33:48.849909+0000","last_active":"2024-03-16T20:34:51.907483+0000","last_peered":"2024-03-16T20:34:51.907483+0000","last_clean":"2024-03-16T20:34:51.907483+0000","last_became_active":"2024-03-16T20:33:48.848645+0000","last_became_peered":"2024-03-16T20:33:48.848645+0000","last_unstale":"2024-03-16T20:34:51.907483+0000","last_undegraded":"2024-03-16T20:34:51.907483+0000","last_fullsized":"2024-03-16T20:34:51.907483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:44.401679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.992869+0000","last_change":"2024-03-16T20:33:49.853100+0000","last_active":"2024-03-16T20:35:19.992869+0000","last_peered":"2024-03-16T20:35:19.992869+0000","last_clean":"2024-03-16T20:35:19.992869+0000","last_became_active":"2024-03-16T20:33:49.852743+0000","last_became_peered":"2024-03-16T20:33:49.852743+0000","last_unstale":"2024-03-16T20:35:19.992869+0000","last_undegraded":"2024-03-16T20:35:19.992869+0000","last_fullsized":"2024-03-16T20:35:19.992869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:42.664026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412319+0000","last_change":"2024-03-16T20:33:53.881626+0000","last_active":"2024-03-16T20:34:51.412319+0000","last_peered":"2024-03-16T20:34:51.412319+0000","last_clean":"2024-03-16T20:34:51.412319+0000","last_became_active":"2024-03-16T20:33:53.881205+0000","last_became_peered":"2024-03-16T20:33:53.881205+0000","last_unstale":"2024-03-16T20:34:51.412319+0000","last_undegraded":"2024-03-16T20:34:51.412319+0000","last_fullsized":"2024-03-16T20:34:51.412319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:54:19.525953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'35","reported_seq":217,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:21.004380+0000","last_change":"2024-03-16T20:33:48.848490+0000","last_active":"2024-03-16T20:35:21.004380+0000","last_peered":"2024-03-16T20:35:21.004380+0000","last_clean":"2024-03-16T20:35:21.004380+0000","last_became_active":"2024-03-16T20:33:48.848152+0000","last_became_peered":"2024-03-16T20:33:48.848152+0000","last_unstale":"2024-03-16T20:35:21.004380+0000","last_undegraded":"2024-03-16T20:35:21.004380+0000","last_fullsized":"2024-03-16T20:35:21.004380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:43:19.908178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":155,"num_read_kb":138,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.955150+0000","last_change":"2024-03-16T20:33:49.851269+0000","last_active":"2024-03-16T20:35:19.955150+0000","last_peered":"2024-03-16T20:35:19.955150+0000","last_clean":"2024-03-16T20:35:19.955150+0000","last_became_active":"2024-03-16T20:33:49.850963+0000","last_became_peered":"2024-03-16T20:33:49.850963+0000","last_unstale":"2024-03-16T20:35:19.955150+0000","last_undegraded":"2024-03-16T20:35:19.955150+0000","last_fullsized":"2024-03-16T20:35:19.955150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:26:01.312043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907845+0000","last_change":"2024-03-16T20:33:53.889419+0000","last_active":"2024-03-16T20:34:51.907845+0000","last_peered":"2024-03-16T20:34:51.907845+0000","last_clean":"2024-03-16T20:34:51.907845+0000","last_became_active":"2024-03-16T20:33:53.889016+0000","last_became_peered":"2024-03-16T20:33:53.889016+0000","last_unstale":"2024-03-16T20:34:51.907845+0000","last_undegraded":"2024-03-16T20:34:51.907845+0000","last_fullsized":"2024-03-16T20:34:51.907845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:32:58.333441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419922+0000","last_change":"2024-03-16T20:33:48.855070+0000","last_active":"2024-03-16T20:34:51.419922+0000","last_peered":"2024-03-16T20:34:51.419922+0000","last_clean":"2024-03-16T20:34:51.419922+0000","last_became_active":"2024-03-16T20:33:48.854669+0000","last_became_peered":"2024-03-16T20:33:48.854669+0000","last_unstale":"2024-03-16T20:34:51.419922+0000","last_undegraded":"2024-03-16T20:34:51.419922+0000","last_fullsized":"2024-03-16T20:34:51.419922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:58:38.229641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983523+0000","last_change":"2024-03-16T20:33:49.850186+0000","last_active":"2024-03-16T20:35:19.983523+0000","last_peered":"2024-03-16T20:35:19.983523+0000","last_clean":"2024-03-16T20:35:19.983523+0000","last_became_active":"2024-03-16T20:33:49.849811+0000","last_became_peered":"2024-03-16T20:33:49.849811+0000","last_unstale":"2024-03-16T20:35:19.983523+0000","last_undegraded":"2024-03-16T20:35:19.983523+0000","last_fullsized":"2024-03-16T20:35:19.983523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:13:03.399984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415127+0000","last_change":"2024-03-16T20:33:53.890030+0000","last_active":"2024-03-16T20:34:51.415127+0000","last_peered":"2024-03-16T20:34:51.415127+0000","last_clean":"2024-03-16T20:34:51.415127+0000","last_became_active":"2024-03-16T20:33:53.889602+0000","last_became_peered":"2024-03-16T20:33:53.889602+0000","last_unstale":"2024-03-16T20:34:51.415127+0000","last_undegraded":"2024-03-16T20:34:51.415127+0000","last_fullsized":"2024-03-16T20:34:51.415127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:04.319150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907456+0000","last_change":"2024-03-16T20:33:48.840422+0000","last_active":"2024-03-16T20:34:51.907456+0000","last_peered":"2024-03-16T20:34:51.907456+0000","last_clean":"2024-03-16T20:34:51.907456+0000","last_became_active":"2024-03-16T20:33:48.840125+0000","last_became_peered":"2024-03-16T20:33:48.840125+0000","last_unstale":"2024-03-16T20:34:51.907456+0000","last_undegraded":"2024-03-16T20:34:51.907456+0000","last_fullsized":"2024-03-16T20:34:51.907456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:29:05.397252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418151+0000","last_change":"2024-03-16T20:33:49.862475+0000","last_active":"2024-03-16T20:34:51.418151+0000","last_peered":"2024-03-16T20:34:51.418151+0000","last_clean":"2024-03-16T20:34:51.418151+0000","last_became_active":"2024-03-16T20:33:49.862032+0000","last_became_peered":"2024-03-16T20:33:49.862032+0000","last_unstale":"2024-03-16T20:34:51.418151+0000","last_undegraded":"2024-03-16T20:34:51.418151+0000","last_fullsized":"2024-03-16T20:34:51.418151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:16:53.319352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412513+0000","last_change":"2024-03-16T20:33:53.884019+0000","last_active":"2024-03-16T20:34:51.412513+0000","last_peered":"2024-03-16T20:34:51.412513+0000","last_clean":"2024-03-16T20:34:51.412513+0000","last_became_active":"2024-03-16T20:33:53.883608+0000","last_became_peered":"2024-03-16T20:33:53.883608+0000","last_unstale":"2024-03-16T20:34:51.412513+0000","last_undegraded":"2024-03-16T20:34:51.412513+0000","last_fullsized":"2024-03-16T20:34:51.412513+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:13:58.689682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":105,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:18.002758+0000","last_change":"2024-03-16T20:33:48.848145+0000","last_active":"2024-03-16T20:35:18.002758+0000","last_peered":"2024-03-16T20:35:18.002758+0000","last_clean":"2024-03-16T20:35:18.002758+0000","last_became_active":"2024-03-16T20:33:48.847754+0000","last_became_peered":"2024-03-16T20:33:48.847754+0000","last_unstale":"2024-03-16T20:35:18.002758+0000","last_undegraded":"2024-03-16T20:35:18.002758+0000","last_fullsized":"2024-03-16T20:35:18.002758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:59.967346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417966+0000","last_change":"2024-03-16T20:33:49.851307+0000","last_active":"2024-03-16T20:34:51.417966+0000","last_peered":"2024-03-16T20:34:51.417966+0000","last_clean":"2024-03-16T20:34:51.417966+0000","last_became_active":"2024-03-16T20:33:49.850930+0000","last_became_peered":"2024-03-16T20:33:49.850930+0000","last_unstale":"2024-03-16T20:34:51.417966+0000","last_undegraded":"2024-03-16T20:34:51.417966+0000","last_fullsized":"2024-03-16T20:34:51.417966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:23:54.528895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416343+0000","last_change":"2024-03-16T20:33:53.893149+0000","last_active":"2024-03-16T20:34:51.416343+0000","last_peered":"2024-03-16T20:34:51.416343+0000","last_clean":"2024-03-16T20:34:51.416343+0000","last_became_active":"2024-03-16T20:33:53.892532+0000","last_became_peered":"2024-03-16T20:33:53.892532+0000","last_unstale":"2024-03-16T20:34:51.416343+0000","last_undegraded":"2024-03-16T20:34:51.416343+0000","last_fullsized":"2024-03-16T20:34:51.416343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:09:09.408974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412677+0000","last_change":"2024-03-16T20:33:48.849788+0000","last_active":"2024-03-16T20:34:51.412677+0000","last_peered":"2024-03-16T20:34:51.412677+0000","last_clean":"2024-03-16T20:34:51.412677+0000","last_became_active":"2024-03-16T20:33:48.849360+0000","last_became_peered":"2024-03-16T20:33:48.849360+0000","last_unstale":"2024-03-16T20:34:51.412677+0000","last_undegraded":"2024-03-16T20:34:51.412677+0000","last_fullsized":"2024-03-16T20:34:51.412677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:53:29.026216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907448+0000","last_change":"2024-03-16T20:33:49.856702+0000","last_active":"2024-03-16T20:34:51.907448+0000","last_peered":"2024-03-16T20:34:51.907448+0000","last_clean":"2024-03-16T20:34:51.907448+0000","last_became_active":"2024-03-16T20:33:49.856373+0000","last_became_peered":"2024-03-16T20:33:49.856373+0000","last_unstale":"2024-03-16T20:34:51.907448+0000","last_undegraded":"2024-03-16T20:34:51.907448+0000","last_fullsized":"2024-03-16T20:34:51.907448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:31:40.093440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907350+0000","last_change":"2024-03-16T20:33:53.884193+0000","last_active":"2024-03-16T20:34:51.907350+0000","last_peered":"2024-03-16T20:34:51.907350+0000","last_clean":"2024-03-16T20:34:51.907350+0000","last_became_active":"2024-03-16T20:33:53.883763+0000","last_became_peered":"2024-03-16T20:33:53.883763+0000","last_unstale":"2024-03-16T20:34:51.907350+0000","last_undegraded":"2024-03-16T20:34:51.907350+0000","last_fullsized":"2024-03-16T20:34:51.907350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:50:20.969662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413379+0000","last_change":"2024-03-16T20:33:48.851166+0000","last_active":"2024-03-16T20:34:51.413379+0000","last_peered":"2024-03-16T20:34:51.413379+0000","last_clean":"2024-03-16T20:34:51.413379+0000","last_became_active":"2024-03-16T20:33:48.850843+0000","last_became_peered":"2024-03-16T20:33:48.850843+0000","last_unstale":"2024-03-16T20:34:51.413379+0000","last_undegraded":"2024-03-16T20:34:51.413379+0000","last_fullsized":"2024-03-16T20:34:51.413379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:06:09.142942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979966+0000","last_change":"2024-03-16T20:33:49.860038+0000","last_active":"2024-03-16T20:35:19.979966+0000","last_peered":"2024-03-16T20:35:19.979966+0000","last_clean":"2024-03-16T20:35:19.979966+0000","last_became_active":"2024-03-16T20:33:49.859715+0000","last_became_peered":"2024-03-16T20:33:49.859715+0000","last_unstale":"2024-03-16T20:35:19.979966+0000","last_undegraded":"2024-03-16T20:35:19.979966+0000","last_fullsized":"2024-03-16T20:35:19.979966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:27:32.800166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413095+0000","last_change":"2024-03-16T20:33:53.930129+0000","last_active":"2024-03-16T20:34:51.413095+0000","last_peered":"2024-03-16T20:34:51.413095+0000","last_clean":"2024-03-16T20:34:51.413095+0000","last_became_active":"2024-03-16T20:33:53.929655+0000","last_became_peered":"2024-03-16T20:33:53.929655+0000","last_unstale":"2024-03-16T20:34:51.413095+0000","last_undegraded":"2024-03-16T20:34:51.413095+0000","last_fullsized":"2024-03-16T20:34:51.413095+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:46.422720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412652+0000","last_change":"2024-03-16T20:33:48.849089+0000","last_active":"2024-03-16T20:34:51.412652+0000","last_peered":"2024-03-16T20:34:51.412652+0000","last_clean":"2024-03-16T20:34:51.412652+0000","last_became_active":"2024-03-16T20:33:48.848719+0000","last_became_peered":"2024-03-16T20:33:48.848719+0000","last_unstale":"2024-03-16T20:34:51.412652+0000","last_undegraded":"2024-03-16T20:34:51.412652+0000","last_fullsized":"2024-03-16T20:34:51.412652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:31:23.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.962814+0000","last_change":"2024-03-16T20:33:49.853247+0000","last_active":"2024-03-16T20:35:19.962814+0000","last_peered":"2024-03-16T20:35:19.962814+0000","last_clean":"2024-03-16T20:35:19.962814+0000","last_became_active":"2024-03-16T20:33:49.852870+0000","last_became_peered":"2024-03-16T20:33:49.852870+0000","last_unstale":"2024-03-16T20:35:19.962814+0000","last_undegraded":"2024-03-16T20:35:19.962814+0000","last_fullsized":"2024-03-16T20:35:19.962814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:48:56.365484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416383+0000","last_change":"2024-03-16T20:33:53.887350+0000","last_active":"2024-03-16T20:34:51.416383+0000","last_peered":"2024-03-16T20:34:51.416383+0000","last_clean":"2024-03-16T20:34:51.416383+0000","last_became_active":"2024-03-16T20:33:53.886782+0000","last_became_peered":"2024-03-16T20:33:53.886782+0000","last_unstale":"2024-03-16T20:34:51.416383+0000","last_undegraded":"2024-03-16T20:34:51.416383+0000","last_fullsized":"2024-03-16T20:34:51.416383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:00:39.898430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":102,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:18.035444+0000","last_change":"2024-03-16T20:33:48.851543+0000","last_active":"2024-03-16T20:35:18.035444+0000","last_peered":"2024-03-16T20:35:18.035444+0000","last_clean":"2024-03-16T20:35:18.035444+0000","last_became_active":"2024-03-16T20:33:48.851196+0000","last_became_peered":"2024-03-16T20:33:48.851196+0000","last_unstale":"2024-03-16T20:35:18.035444+0000","last_undegraded":"2024-03-16T20:35:18.035444+0000","last_fullsized":"2024-03-16T20:35:18.035444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:06.309259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.962510+0000","last_change":"2024-03-16T20:33:49.856021+0000","last_active":"2024-03-16T20:35:19.962510+0000","last_peered":"2024-03-16T20:35:19.962510+0000","last_clean":"2024-03-16T20:35:19.962510+0000","last_became_active":"2024-03-16T20:33:49.854130+0000","last_became_peered":"2024-03-16T20:33:49.854130+0000","last_unstale":"2024-03-16T20:35:19.962510+0000","last_undegraded":"2024-03-16T20:35:19.962510+0000","last_fullsized":"2024-03-16T20:35:19.962510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:29:39.970391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416561+0000","last_change":"2024-03-16T20:33:53.930094+0000","last_active":"2024-03-16T20:34:51.416561+0000","last_peered":"2024-03-16T20:34:51.416561+0000","last_clean":"2024-03-16T20:34:51.416561+0000","last_became_active":"2024-03-16T20:33:53.929613+0000","last_became_peered":"2024-03-16T20:33:53.929613+0000","last_unstale":"2024-03-16T20:34:51.416561+0000","last_undegraded":"2024-03-16T20:34:51.416561+0000","last_fullsized":"2024-03-16T20:34:51.416561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:30.680295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907471+0000","last_change":"2024-03-16T20:33:49.857949+0000","last_active":"2024-03-16T20:34:51.907471+0000","last_peered":"2024-03-16T20:34:51.907471+0000","last_clean":"2024-03-16T20:34:51.907471+0000","last_became_active":"2024-03-16T20:33:49.857642+0000","last_became_peered":"2024-03-16T20:33:49.857642+0000","last_unstale":"2024-03-16T20:34:51.907471+0000","last_undegraded":"2024-03-16T20:34:51.907471+0000","last_fullsized":"2024-03-16T20:34:51.907471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:23.040787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":105,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.003384+0000","last_change":"2024-03-16T20:33:48.856908+0000","last_active":"2024-03-16T20:35:19.003384+0000","last_peered":"2024-03-16T20:35:19.003384+0000","last_clean":"2024-03-16T20:35:19.003384+0000","last_became_active":"2024-03-16T20:33:48.856578+0000","last_became_peered":"2024-03-16T20:33:48.856578+0000","last_unstale":"2024-03-16T20:35:19.003384+0000","last_undegraded":"2024-03-16T20:35:19.003384+0000","last_fullsized":"2024-03-16T20:35:19.003384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:55:37.467052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418818+0000","last_change":"2024-03-16T20:33:53.881965+0000","last_active":"2024-03-16T20:34:51.418818+0000","last_peered":"2024-03-16T20:34:51.418818+0000","last_clean":"2024-03-16T20:34:51.418818+0000","last_became_active":"2024-03-16T20:33:53.881247+0000","last_became_peered":"2024-03-16T20:33:53.881247+0000","last_unstale":"2024-03-16T20:34:51.418818+0000","last_undegraded":"2024-03-16T20:34:51.418818+0000","last_fullsized":"2024-03-16T20:34:51.418818+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:54:18.868143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.957596+0000","last_change":"2024-03-16T20:34:51.909039+0000","last_active":"2024-03-16T20:35:19.957596+0000","last_peered":"2024-03-16T20:35:19.957596+0000","last_clean":"2024-03-16T20:35:19.957596+0000","last_became_active":"2024-03-16T20:34:51.908628+0000","last_became_peered":"2024-03-16T20:34:51.908628+0000","last_unstale":"2024-03-16T20:35:19.957596+0000","last_undegraded":"2024-03-16T20:35:19.957596+0000","last_fullsized":"2024-03-16T20:35:19.957596+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:29:02.007925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415231+0000","last_change":"2024-03-16T20:33:48.850177+0000","last_active":"2024-03-16T20:34:51.415231+0000","last_peered":"2024-03-16T20:34:51.415231+0000","last_clean":"2024-03-16T20:34:51.415231+0000","last_became_active":"2024-03-16T20:33:48.849838+0000","last_became_peered":"2024-03-16T20:33:48.849838+0000","last_unstale":"2024-03-16T20:34:51.415231+0000","last_undegraded":"2024-03-16T20:34:51.415231+0000","last_fullsized":"2024-03-16T20:34:51.415231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:28:24.544880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417808+0000","last_change":"2024-03-16T20:32:34.913833+0000","last_active":"2024-03-16T20:34:51.417808+0000","last_peered":"2024-03-16T20:34:51.417808+0000","last_clean":"2024-03-16T20:34:51.417808+0000","last_became_active":"2024-03-16T20:32:34.913443+0000","last_became_peered":"2024-03-16T20:32:34.913443+0000","last_unstale":"2024-03-16T20:34:51.417808+0000","last_undegraded":"2024-03-16T20:34:51.417808+0000","last_fullsized":"2024-03-16T20:34:51.417808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:29:10.760383+0000","last_clean_scrub_stamp":"2024-03-16T20:29:10.760383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:37:05.333386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907623+0000","last_change":"2024-03-16T20:33:53.931311+0000","last_active":"2024-03-16T20:34:51.907623+0000","last_peered":"2024-03-16T20:34:51.907623+0000","last_clean":"2024-03-16T20:34:51.907623+0000","last_became_active":"2024-03-16T20:33:53.930972+0000","last_became_peered":"2024-03-16T20:33:53.930972+0000","last_unstale":"2024-03-16T20:34:51.907623+0000","last_undegraded":"2024-03-16T20:34:51.907623+0000","last_fullsized":"2024-03-16T20:34:51.907623+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:03:29.776651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.981100+0000","last_change":"2024-03-16T20:33:49.838866+0000","last_active":"2024-03-16T20:35:19.981100+0000","last_peered":"2024-03-16T20:35:19.981100+0000","last_clean":"2024-03-16T20:35:19.981100+0000","last_became_active":"2024-03-16T20:33:49.838561+0000","last_became_peered":"2024-03-16T20:33:49.838561+0000","last_unstale":"2024-03-16T20:35:19.981100+0000","last_undegraded":"2024-03-16T20:35:19.981100+0000","last_fullsized":"2024-03-16T20:35:19.981100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:57:21.098078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417865+0000","last_change":"2024-03-16T20:33:48.847098+0000","last_active":"2024-03-16T20:34:51.417865+0000","last_peered":"2024-03-16T20:34:51.417865+0000","last_clean":"2024-03-16T20:34:51.417865+0000","last_became_active":"2024-03-16T20:33:48.846399+0000","last_became_peered":"2024-03-16T20:33:48.846399+0000","last_unstale":"2024-03-16T20:34:51.417865+0000","last_undegraded":"2024-03-16T20:34:51.417865+0000","last_fullsized":"2024-03-16T20:34:51.417865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:41:59.126547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415394+0000","last_change":"2024-03-16T20:33:53.890952+0000","last_active":"2024-03-16T20:34:51.415394+0000","last_peered":"2024-03-16T20:34:51.415394+0000","last_clean":"2024-03-16T20:34:51.415394+0000","last_became_active":"2024-03-16T20:33:53.890473+0000","last_became_peered":"2024-03-16T20:33:53.890473+0000","last_unstale":"2024-03-16T20:34:51.415394+0000","last_undegraded":"2024-03-16T20:34:51.415394+0000","last_fullsized":"2024-03-16T20:34:51.415394+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:06.120997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.413602+0000","last_change":"2024-03-16T20:33:48.851220+0000","last_active":"2024-03-16T20:34:51.413602+0000","last_peered":"2024-03-16T20:34:51.413602+0000","last_clean":"2024-03-16T20:34:51.413602+0000","last_became_active":"2024-03-16T20:33:48.850927+0000","last_became_peered":"2024-03-16T20:33:48.850927+0000","last_unstale":"2024-03-16T20:34:51.413602+0000","last_undegraded":"2024-03-16T20:34:51.413602+0000","last_fullsized":"2024-03-16T20:34:51.413602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:40:50.105276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983955+0000","last_change":"2024-03-16T20:33:49.850272+0000","last_active":"2024-03-16T20:35:19.983955+0000","last_peered":"2024-03-16T20:35:19.983955+0000","last_clean":"2024-03-16T20:35:19.983955+0000","last_became_active":"2024-03-16T20:33:49.849913+0000","last_became_peered":"2024-03-16T20:33:49.849913+0000","last_unstale":"2024-03-16T20:35:19.983955+0000","last_undegraded":"2024-03-16T20:35:19.983955+0000","last_fullsized":"2024-03-16T20:35:19.983955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:53:18.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417469+0000","last_change":"2024-03-16T20:33:53.930421+0000","last_active":"2024-03-16T20:34:51.417469+0000","last_peered":"2024-03-16T20:34:51.417469+0000","last_clean":"2024-03-16T20:34:51.417469+0000","last_became_active":"2024-03-16T20:33:53.929957+0000","last_became_peered":"2024-03-16T20:33:53.929957+0000","last_unstale":"2024-03-16T20:34:51.417469+0000","last_undegraded":"2024-03-16T20:34:51.417469+0000","last_fullsized":"2024-03-16T20:34:51.417469+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:22:20.541592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412961+0000","last_change":"2024-03-16T20:33:48.845096+0000","last_active":"2024-03-16T20:34:51.412961+0000","last_peered":"2024-03-16T20:34:51.412961+0000","last_clean":"2024-03-16T20:34:51.412961+0000","last_became_active":"2024-03-16T20:33:48.844580+0000","last_became_peered":"2024-03-16T20:33:48.844580+0000","last_unstale":"2024-03-16T20:34:51.412961+0000","last_undegraded":"2024-03-16T20:34:51.412961+0000","last_fullsized":"2024-03-16T20:34:51.412961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:56:53.857336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.980180+0000","last_change":"2024-03-16T20:33:49.858488+0000","last_active":"2024-03-16T20:35:19.980180+0000","last_peered":"2024-03-16T20:35:19.980180+0000","last_clean":"2024-03-16T20:35:19.980180+0000","last_became_active":"2024-03-16T20:33:49.858163+0000","last_became_peered":"2024-03-16T20:33:49.858163+0000","last_unstale":"2024-03-16T20:35:19.980180+0000","last_undegraded":"2024-03-16T20:35:19.980180+0000","last_fullsized":"2024-03-16T20:35:19.980180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:23:49.399329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417853+0000","last_change":"2024-03-16T20:33:53.884240+0000","last_active":"2024-03-16T20:34:51.417853+0000","last_peered":"2024-03-16T20:34:51.417853+0000","last_clean":"2024-03-16T20:34:51.417853+0000","last_became_active":"2024-03-16T20:33:53.883864+0000","last_became_peered":"2024-03-16T20:33:53.883864+0000","last_unstale":"2024-03-16T20:34:51.417853+0000","last_undegraded":"2024-03-16T20:34:51.417853+0000","last_fullsized":"2024-03-16T20:34:51.417853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:57:56.073112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415623+0000","last_change":"2024-03-16T20:33:48.858217+0000","last_active":"2024-03-16T20:34:51.415623+0000","last_peered":"2024-03-16T20:34:51.415623+0000","last_clean":"2024-03-16T20:34:51.415623+0000","last_became_active":"2024-03-16T20:33:48.857844+0000","last_became_peered":"2024-03-16T20:33:48.857844+0000","last_unstale":"2024-03-16T20:34:51.415623+0000","last_undegraded":"2024-03-16T20:34:51.415623+0000","last_fullsized":"2024-03-16T20:34:51.415623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:29:09.143250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.972863+0000","last_change":"2024-03-16T20:33:49.856002+0000","last_active":"2024-03-16T20:35:19.972863+0000","last_peered":"2024-03-16T20:35:19.972863+0000","last_clean":"2024-03-16T20:35:19.972863+0000","last_became_active":"2024-03-16T20:33:49.855664+0000","last_became_peered":"2024-03-16T20:33:49.855664+0000","last_unstale":"2024-03-16T20:35:19.972863+0000","last_undegraded":"2024-03-16T20:35:19.972863+0000","last_fullsized":"2024-03-16T20:35:19.972863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:55:36.502729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.761360+0000","last_change":"2024-03-16T20:34:51.761360+0000","last_active":"2024-03-16T20:34:51.761360+0000","last_peered":"2024-03-16T20:34:51.761360+0000","last_clean":"2024-03-16T20:34:51.761360+0000","last_became_active":"2024-03-16T20:34:51.760903+0000","last_became_peered":"2024-03-16T20:34:51.760903+0000","last_unstale":"2024-03-16T20:34:51.761360+0000","last_undegraded":"2024-03-16T20:34:51.761360+0000","last_fullsized":"2024-03-16T20:34:51.761360+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:53:08.015778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907519+0000","last_change":"2024-03-16T20:33:48.855853+0000","last_active":"2024-03-16T20:34:51.907519+0000","last_peered":"2024-03-16T20:34:51.907519+0000","last_clean":"2024-03-16T20:34:51.907519+0000","last_became_active":"2024-03-16T20:33:48.855539+0000","last_became_peered":"2024-03-16T20:33:48.855539+0000","last_unstale":"2024-03-16T20:34:51.907519+0000","last_undegraded":"2024-03-16T20:34:51.907519+0000","last_fullsized":"2024-03-16T20:34:51.907519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:56:48.618350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.935552+0000","last_change":"2024-03-16T20:33:49.854825+0000","last_active":"2024-03-16T20:35:19.935552+0000","last_peered":"2024-03-16T20:35:19.935552+0000","last_clean":"2024-03-16T20:35:19.935552+0000","last_became_active":"2024-03-16T20:33:49.854396+0000","last_became_peered":"2024-03-16T20:33:49.854396+0000","last_unstale":"2024-03-16T20:35:19.935552+0000","last_undegraded":"2024-03-16T20:35:19.935552+0000","last_fullsized":"2024-03-16T20:35:19.935552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:31:18.133694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.412944+0000","last_change":"2024-03-16T20:33:53.881554+0000","last_active":"2024-03-16T20:34:51.412944+0000","last_peered":"2024-03-16T20:34:51.412944+0000","last_clean":"2024-03-16T20:34:51.412944+0000","last_became_active":"2024-03-16T20:33:53.881030+0000","last_became_peered":"2024-03-16T20:33:53.881030+0000","last_unstale":"2024-03-16T20:34:51.412944+0000","last_undegraded":"2024-03-16T20:34:51.412944+0000","last_fullsized":"2024-03-16T20:34:51.412944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:08:46.499466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.417928+0000","last_change":"2024-03-16T20:33:48.853138+0000","last_active":"2024-03-16T20:34:51.417928+0000","last_peered":"2024-03-16T20:34:51.417928+0000","last_clean":"2024-03-16T20:34:51.417928+0000","last_became_active":"2024-03-16T20:33:48.851501+0000","last_became_peered":"2024-03-16T20:33:48.851501+0000","last_unstale":"2024-03-16T20:34:51.417928+0000","last_undegraded":"2024-03-16T20:34:51.417928+0000","last_fullsized":"2024-03-16T20:34:51.417928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:57:40.368614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.982482+0000","last_change":"2024-03-16T20:33:49.850307+0000","last_active":"2024-03-16T20:35:19.982482+0000","last_peered":"2024-03-16T20:35:19.982482+0000","last_clean":"2024-03-16T20:35:19.982482+0000","last_became_active":"2024-03-16T20:33:49.849953+0000","last_became_peered":"2024-03-16T20:33:49.849953+0000","last_unstale":"2024-03-16T20:35:19.982482+0000","last_undegraded":"2024-03-16T20:35:19.982482+0000","last_fullsized":"2024-03-16T20:35:19.982482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:36:51.020011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415217+0000","last_change":"2024-03-16T20:33:53.892370+0000","last_active":"2024-03-16T20:34:51.415217+0000","last_peered":"2024-03-16T20:34:51.415217+0000","last_clean":"2024-03-16T20:34:51.415217+0000","last_became_active":"2024-03-16T20:33:53.891885+0000","last_became_peered":"2024-03-16T20:33:53.891885+0000","last_unstale":"2024-03-16T20:34:51.415217+0000","last_undegraded":"2024-03-16T20:34:51.415217+0000","last_fullsized":"2024-03-16T20:34:51.415217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:55:30.770882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.425434+0000","last_change":"2024-03-16T20:34:51.424981+0000","last_active":"2024-03-16T20:34:51.425434+0000","last_peered":"2024-03-16T20:34:51.425434+0000","last_clean":"2024-03-16T20:34:51.425434+0000","last_became_active":"2024-03-16T20:34:51.420101+0000","last_became_peered":"2024-03-16T20:34:51.420101+0000","last_unstale":"2024-03-16T20:34:51.425434+0000","last_undegraded":"2024-03-16T20:34:51.425434+0000","last_fullsized":"2024-03-16T20:34:51.425434+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:21:32.567770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.980271+0000","last_change":"2024-03-16T20:33:49.861637+0000","last_active":"2024-03-16T20:35:19.980271+0000","last_peered":"2024-03-16T20:35:19.980271+0000","last_clean":"2024-03-16T20:35:19.980271+0000","last_became_active":"2024-03-16T20:33:49.861284+0000","last_became_peered":"2024-03-16T20:33:49.861284+0000","last_unstale":"2024-03-16T20:35:19.980271+0000","last_undegraded":"2024-03-16T20:35:19.980271+0000","last_fullsized":"2024-03-16T20:35:19.980271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:01:30.937928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.415270+0000","last_change":"2024-03-16T20:33:53.931633+0000","last_active":"2024-03-16T20:34:51.415270+0000","last_peered":"2024-03-16T20:34:51.415270+0000","last_clean":"2024-03-16T20:34:51.415270+0000","last_became_active":"2024-03-16T20:33:53.930633+0000","last_became_peered":"2024-03-16T20:33:53.930633+0000","last_unstale":"2024-03-16T20:34:51.415270+0000","last_undegraded":"2024-03-16T20:34:51.415270+0000","last_fullsized":"2024-03-16T20:34:51.415270+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:33:57.356317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.429155+0000","last_change":"2024-03-16T20:34:51.428773+0000","last_active":"2024-03-16T20:34:51.429155+0000","last_peered":"2024-03-16T20:34:51.429155+0000","last_clean":"2024-03-16T20:34:51.429155+0000","last_became_active":"2024-03-16T20:34:51.420784+0000","last_became_peered":"2024-03-16T20:34:51.420784+0000","last_unstale":"2024-03-16T20:34:51.429155+0000","last_undegraded":"2024-03-16T20:34:51.429155+0000","last_fullsized":"2024-03-16T20:34:51.429155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:39:54.702810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418483+0000","last_change":"2024-03-16T20:33:49.859085+0000","last_active":"2024-03-16T20:34:51.418483+0000","last_peered":"2024-03-16T20:34:51.418483+0000","last_clean":"2024-03-16T20:34:51.418483+0000","last_became_active":"2024-03-16T20:33:49.858637+0000","last_became_peered":"2024-03-16T20:33:49.858637+0000","last_unstale":"2024-03-16T20:34:51.418483+0000","last_undegraded":"2024-03-16T20:34:51.418483+0000","last_fullsized":"2024-03-16T20:34:51.418483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:22:17.020669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907388+0000","last_change":"2024-03-16T20:33:53.891866+0000","last_active":"2024-03-16T20:34:51.907388+0000","last_peered":"2024-03-16T20:34:51.907388+0000","last_clean":"2024-03-16T20:34:51.907388+0000","last_became_active":"2024-03-16T20:33:53.891424+0000","last_became_peered":"2024-03-16T20:33:53.891424+0000","last_unstale":"2024-03-16T20:34:51.907388+0000","last_undegraded":"2024-03-16T20:34:51.907388+0000","last_fullsized":"2024-03-16T20:34:51.907388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T04:25:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.036662+0000","last_change":"2024-03-16T20:33:48.856124+0000","last_active":"2024-03-16T20:35:19.036662+0000","last_peered":"2024-03-16T20:35:19.036662+0000","last_clean":"2024-03-16T20:35:19.036662+0000","last_became_active":"2024-03-16T20:33:48.855813+0000","last_became_peered":"2024-03-16T20:33:48.855813+0000","last_unstale":"2024-03-16T20:35:19.036662+0000","last_undegraded":"2024-03-16T20:35:19.036662+0000","last_fullsized":"2024-03-16T20:35:19.036662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:05:55.122457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.419046+0000","last_change":"2024-03-16T20:33:49.857479+0000","last_active":"2024-03-16T20:34:51.419046+0000","last_peered":"2024-03-16T20:34:51.419046+0000","last_clean":"2024-03-16T20:34:51.419046+0000","last_became_active":"2024-03-16T20:33:49.856121+0000","last_became_peered":"2024-03-16T20:33:49.856121+0000","last_unstale":"2024-03-16T20:34:51.419046+0000","last_undegraded":"2024-03-16T20:34:51.419046+0000","last_fullsized":"2024-03-16T20:34:51.419046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:08:55.613977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.909071+0000","last_change":"2024-03-16T20:34:51.909071+0000","last_active":"2024-03-16T20:34:51.909071+0000","last_peered":"2024-03-16T20:34:51.909071+0000","last_clean":"2024-03-16T20:34:51.909071+0000","last_became_active":"2024-03-16T20:34:51.908570+0000","last_became_peered":"2024-03-16T20:34:51.908570+0000","last_unstale":"2024-03-16T20:34:51.909071+0000","last_undegraded":"2024-03-16T20:34:51.909071+0000","last_fullsized":"2024-03-16T20:34:51.909071+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:19:18.023401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,1,4],"acting":[0,10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907708+0000","last_change":"2024-03-16T20:33:48.846628+0000","last_active":"2024-03-16T20:34:51.907708+0000","last_peered":"2024-03-16T20:34:51.907708+0000","last_clean":"2024-03-16T20:34:51.907708+0000","last_became_active":"2024-03-16T20:33:48.846276+0000","last_became_peered":"2024-03-16T20:33:48.846276+0000","last_unstale":"2024-03-16T20:34:51.907708+0000","last_undegraded":"2024-03-16T20:34:51.907708+0000","last_fullsized":"2024-03-16T20:34:51.907708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:24.323938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418252+0000","last_change":"2024-03-16T20:33:49.854041+0000","last_active":"2024-03-16T20:34:51.418252+0000","last_peered":"2024-03-16T20:34:51.418252+0000","last_clean":"2024-03-16T20:34:51.418252+0000","last_became_active":"2024-03-16T20:33:49.853653+0000","last_became_peered":"2024-03-16T20:33:49.853653+0000","last_unstale":"2024-03-16T20:34:51.418252+0000","last_undegraded":"2024-03-16T20:34:51.418252+0000","last_fullsized":"2024-03-16T20:34:51.418252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:59:10.498314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.906967+0000","last_change":"2024-03-16T20:33:53.884292+0000","last_active":"2024-03-16T20:34:51.906967+0000","last_peered":"2024-03-16T20:34:51.906967+0000","last_clean":"2024-03-16T20:34:51.906967+0000","last_became_active":"2024-03-16T20:33:53.883778+0000","last_became_peered":"2024-03-16T20:33:53.883778+0000","last_unstale":"2024-03-16T20:34:51.906967+0000","last_undegraded":"2024-03-16T20:34:51.906967+0000","last_fullsized":"2024-03-16T20:34:51.906967+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:52.849643+0000","last_clean_scrub_stamp":"2024-03-16T20:33:52.849643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:38:33.877142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.416450+0000","last_change":"2024-03-16T20:33:48.845277+0000","last_active":"2024-03-16T20:34:51.416450+0000","last_peered":"2024-03-16T20:34:51.416450+0000","last_clean":"2024-03-16T20:34:51.416450+0000","last_became_active":"2024-03-16T20:33:48.844954+0000","last_became_peered":"2024-03-16T20:33:48.844954+0000","last_unstale":"2024-03-16T20:34:51.416450+0000","last_undegraded":"2024-03-16T20:34:51.416450+0000","last_fullsized":"2024-03-16T20:34:51.416450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:47.818187+0000","last_clean_scrub_stamp":"2024-03-16T20:33:47.818187+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:24:59.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.418737+0000","last_change":"2024-03-16T20:33:49.856152+0000","last_active":"2024-03-16T20:34:51.418737+0000","last_peered":"2024-03-16T20:34:51.418737+0000","last_clean":"2024-03-16T20:34:51.418737+0000","last_became_active":"2024-03-16T20:33:49.855026+0000","last_became_peered":"2024-03-16T20:33:49.855026+0000","last_unstale":"2024-03-16T20:34:51.418737+0000","last_undegraded":"2024-03-16T20:34:51.418737+0000","last_fullsized":"2024-03-16T20:34:51.418737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:31:40.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.980547+0000","last_change":"2024-03-16T20:33:49.851539+0000","last_active":"2024-03-16T20:35:19.980547+0000","last_peered":"2024-03-16T20:35:19.980547+0000","last_clean":"2024-03-16T20:35:19.980547+0000","last_became_active":"2024-03-16T20:33:49.851188+0000","last_became_peered":"2024-03-16T20:33:49.851188+0000","last_unstale":"2024-03-16T20:35:19.980547+0000","last_undegraded":"2024-03-16T20:35:19.980547+0000","last_fullsized":"2024-03-16T20:35:19.980547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:34:00.153832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976313+0000","last_change":"2024-03-16T20:33:49.845150+0000","last_active":"2024-03-16T20:35:19.976313+0000","last_peered":"2024-03-16T20:35:19.976313+0000","last_clean":"2024-03-16T20:35:19.976313+0000","last_became_active":"2024-03-16T20:33:49.844531+0000","last_became_peered":"2024-03-16T20:33:49.844531+0000","last_unstale":"2024-03-16T20:35:19.976313+0000","last_undegraded":"2024-03-16T20:35:19.976313+0000","last_fullsized":"2024-03-16T20:35:19.976313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:46:54.607559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.970119+0000","last_change":"2024-03-16T20:33:49.853608+0000","last_active":"2024-03-16T20:35:19.970119+0000","last_peered":"2024-03-16T20:35:19.970119+0000","last_clean":"2024-03-16T20:35:19.970119+0000","last_became_active":"2024-03-16T20:33:49.852800+0000","last_became_peered":"2024-03-16T20:33:49.852800+0000","last_unstale":"2024-03-16T20:35:19.970119+0000","last_undegraded":"2024-03-16T20:35:19.970119+0000","last_fullsized":"2024-03-16T20:35:19.970119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:34:59.988313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.978276+0000","last_change":"2024-03-16T20:33:49.863542+0000","last_active":"2024-03-16T20:35:19.978276+0000","last_peered":"2024-03-16T20:35:19.978276+0000","last_clean":"2024-03-16T20:35:19.978276+0000","last_became_active":"2024-03-16T20:33:49.863143+0000","last_became_peered":"2024-03-16T20:33:49.863143+0000","last_unstale":"2024-03-16T20:35:19.978276+0000","last_undegraded":"2024-03-16T20:35:19.978276+0000","last_fullsized":"2024-03-16T20:35:19.978276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T08:18:29.997097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.984190+0000","last_change":"2024-03-16T20:34:51.418944+0000","last_active":"2024-03-16T20:35:19.984190+0000","last_peered":"2024-03-16T20:35:19.984190+0000","last_clean":"2024-03-16T20:35:19.984190+0000","last_became_active":"2024-03-16T20:34:51.418559+0000","last_became_peered":"2024-03-16T20:34:51.418559+0000","last_unstale":"2024-03-16T20:35:19.984190+0000","last_undegraded":"2024-03-16T20:35:19.984190+0000","last_fullsized":"2024-03-16T20:35:19.984190+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T03:59:27.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.977322+0000","last_change":"2024-03-16T20:33:49.853004+0000","last_active":"2024-03-16T20:35:19.977322+0000","last_peered":"2024-03-16T20:35:19.977322+0000","last_clean":"2024-03-16T20:35:19.977322+0000","last_became_active":"2024-03-16T20:33:49.852609+0000","last_became_peered":"2024-03-16T20:33:49.852609+0000","last_unstale":"2024-03-16T20:35:19.977322+0000","last_undegraded":"2024-03-16T20:35:19.977322+0000","last_fullsized":"2024-03-16T20:35:19.977322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T05:42:58.648008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.989803+0000","last_change":"2024-03-16T20:33:49.857259+0000","last_active":"2024-03-16T20:35:19.989803+0000","last_peered":"2024-03-16T20:35:19.989803+0000","last_clean":"2024-03-16T20:35:19.989803+0000","last_became_active":"2024-03-16T20:33:49.856898+0000","last_became_peered":"2024-03-16T20:33:49.856898+0000","last_unstale":"2024-03-16T20:35:19.989803+0000","last_undegraded":"2024-03-16T20:35:19.989803+0000","last_fullsized":"2024-03-16T20:35:19.989803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:48:54.203804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.976493+0000","last_change":"2024-03-16T20:33:49.860995+0000","last_active":"2024-03-16T20:35:19.976493+0000","last_peered":"2024-03-16T20:35:19.976493+0000","last_clean":"2024-03-16T20:35:19.976493+0000","last_became_active":"2024-03-16T20:33:49.860635+0000","last_became_peered":"2024-03-16T20:33:49.860635+0000","last_unstale":"2024-03-16T20:35:19.976493+0000","last_undegraded":"2024-03-16T20:35:19.976493+0000","last_fullsized":"2024-03-16T20:35:19.976493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T06:28:59.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:34:51.907346+0000","last_change":"2024-03-16T20:33:49.854841+0000","last_active":"2024-03-16T20:34:51.907346+0000","last_peered":"2024-03-16T20:34:51.907346+0000","last_clean":"2024-03-16T20:34:51.907346+0000","last_became_active":"2024-03-16T20:33:49.854468+0000","last_became_peered":"2024-03-16T20:33:49.854468+0000","last_unstale":"2024-03-16T20:34:51.907346+0000","last_undegraded":"2024-03-16T20:34:51.907346+0000","last_fullsized":"2024-03-16T20:34:51.907346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:52.224203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.979399+0000","last_change":"2024-03-16T20:33:49.854757+0000","last_active":"2024-03-16T20:35:19.979399+0000","last_peered":"2024-03-16T20:35:19.979399+0000","last_clean":"2024-03-16T20:35:19.979399+0000","last_became_active":"2024-03-16T20:33:49.852131+0000","last_became_peered":"2024-03-16T20:33:49.852131+0000","last_unstale":"2024-03-16T20:35:19.979399+0000","last_undegraded":"2024-03-16T20:35:19.979399+0000","last_fullsized":"2024-03-16T20:35:19.979399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:41:14.731341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.994849+0000","last_change":"2024-03-16T20:33:49.863332+0000","last_active":"2024-03-16T20:35:19.994849+0000","last_peered":"2024-03-16T20:35:19.994849+0000","last_clean":"2024-03-16T20:35:19.994849+0000","last_became_active":"2024-03-16T20:33:49.862953+0000","last_became_peered":"2024-03-16T20:33:49.862953+0000","last_unstale":"2024-03-16T20:35:19.994849+0000","last_undegraded":"2024-03-16T20:35:19.994849+0000","last_fullsized":"2024-03-16T20:35:19.994849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:42:17.126504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.983077+0000","last_change":"2024-03-16T20:33:49.849055+0000","last_active":"2024-03-16T20:35:19.983077+0000","last_peered":"2024-03-16T20:35:19.983077+0000","last_clean":"2024-03-16T20:35:19.983077+0000","last_became_active":"2024-03-16T20:33:49.848686+0000","last_became_peered":"2024-03-16T20:33:49.848686+0000","last_unstale":"2024-03-16T20:35:19.983077+0000","last_undegraded":"2024-03-16T20:35:19.983077+0000","last_fullsized":"2024-03-16T20:35:19.983077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:57:15.783993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-16T20:35:19.977519+0000","last_change":"2024-03-16T20:34:51.418207+0000","last_active":"2024-03-16T20:35:19.977519+0000","last_peered":"2024-03-16T20:35:19.977519+0000","last_clean":"2024-03-16T20:35:19.977519+0000","last_became_active":"2024-03-16T20:34:51.417875+0000","last_became_peered":"2024-03-16T20:34:51.417875+0000","last_unstale":"2024-03-16T20:35:19.977519+0000","last_undegraded":"2024-03-16T20:35:19.977519+0000","last_fullsized":"2024-03-16T20:35:19.977519+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T20:33:48.825225+0000","last_clean_scrub_stamp":"2024-03-16T20:33:48.825225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T07:46:11.734927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286596,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68284,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":752,"num_read_kb":454,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":204993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":507120,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1155407,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":511146,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":507428,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1103925,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":502554,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":565055,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":504010,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":565417,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":502262,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":506628,"data_compressed":20651,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1099507,"data_compressed":57089,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34802,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36438,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T20:35:26.439 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T20:35:26.439 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T20:35:26.439 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T20:35:26.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T20:35:26.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:26 smithi031 ceph-mon[36281]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:26.440 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:26 smithi031 ceph-mon[36281]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:26 smithi189 ceph-mon[36655]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T20:35:26.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:26 smithi189 ceph-mon[36655]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:28.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:28 smithi023 ceph-mon[31879]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:28.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:28 smithi031 ceph-mon[36281]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:28 smithi189 ceph-mon[36655]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 -- 172.21.15.23:0/3464146861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3068108010 msgr2=0x7f30681083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 --2- 172.21.15.23:0/3464146861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3068108010 0x7f30681083f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f30580099c0 tx=0x7f3058009cd0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 -- 172.21.15.23:0/3464146861 shutdown_connections 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 --2- 172.21.15.23:0/3464146861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3068108010 0x7f30681083f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 --2- 172.21.15.23:0/3464146861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30681029f0 0x7f306810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 --2- 172.21.15.23:0/3464146861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 -- 172.21.15.23:0/3464146861 >> 172.21.15.23:0/3464146861 conn(0x7f3068079110 msgr2=0x7f3068079520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:28.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 -- 172.21.15.23:0/3464146861 shutdown_connections 2024-03-16T20:35:28.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.515+0000 7f306ddcf700 1 -- 172.21.15.23:0/3464146861 wait complete. 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 Processor -- start 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 -- start start 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3068108010 0x7f306819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819d3c0 con 0x7f30681029f0 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819d530 con 0x7f3068102030 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f306ddcf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306819d6f0 con 0x7f3068108010 2024-03-16T20:35:28.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f3066ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f3066ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47926/0 (socket says 172.21.15.23:47926) 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f30677fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f3066ffd700 1 -- 172.21.15.23:0/2082101622 learned_addr learned my addr 172.21.15.23:0/2082101622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.516+0000 7f3067fff700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3068108010 0x7f306819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 -- 172.21.15.23:0/2082101622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3068108010 msgr2=0x7f306819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3068108010 0x7f306819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 -- 172.21.15.23:0/2082101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 msgr2=0x7f30681984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 -- 172.21.15.23:0/2082101622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580095f0 con 0x7f30681029f0 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f30677fe700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3066ffd700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9a0 tx=0x7f305c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:28.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c009790 con 0x7f30681029f0 2024-03-16T20:35:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306819d9d0 con 0x7f30681029f0 2024-03-16T20:35:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f305c010430 con 0x7f30681029f0 2024-03-16T20:35:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c018680 con 0x7f30681029f0 2024-03-16T20:35:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.517+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30680743e0 con 0x7f30681029f0 2024-03-16T20:35:28.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.518+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306810a4f0 con 0x7f30681029f0 2024-03-16T20:35:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.520+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f305c010910 con 0x7f30681029f0 2024-03-16T20:35:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.520+0000 7f3064ff9700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 0x7f30500a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.520+0000 7f30677fe700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 0x7f30500a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.521+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f305c014070 con 0x7f30681029f0 2024-03-16T20:35:28.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.521+0000 7f30677fe700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 0x7f30500a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3058000c00 tx=0x7f305801e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:28.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.524+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f305c096d10 con 0x7f30681029f0 2024-03-16T20:35:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.771+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f30681998a0 con 0x7f30681029f0 2024-03-16T20:35:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.772+0000 7f3064ff9700 1 -- 172.21.15.23:0/2082101622 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f305c0105a0 con 0x7f30681029f0 2024-03-16T20:35:28.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:28.777 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 msgr2=0x7f30500a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 0x7f30500a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3058000c00 tx=0x7f305801e040 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 msgr2=0x7f3068198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9a0 tx=0x7f305c00ecb0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 shutdown_connections 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f30500a1ac0 0x7f30500a3f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3068108010 0x7f306819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3068102030 0x7f30681984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 --2- 172.21.15.23:0/2082101622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30681029f0 0x7f3068198a20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 >> 172.21.15.23:0/2082101622 conn(0x7f3068079110 msgr2=0x7f3068100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:28.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 shutdown_connections 2024-03-16T20:35:28.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:28.774+0000 7f306ddcf700 1 -- 172.21.15.23:0/2082101622 wait complete. 2024-03-16T20:35:29.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:29 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2082101622' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T20:35:29.432 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T20:35:29.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:29.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:29 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2082101622' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T20:35:29.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:29 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2082101622' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T20:35:29.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 -- 172.21.15.23:0/2709152314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 msgr2=0x7f6a48101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 --2- 172.21.15.23:0/2709152314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48101b00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009900 tx=0x7f6a4402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:29.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 -- 172.21.15.23:0/2709152314 shutdown_connections 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 --2- 172.21.15.23:0/2709152314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 --2- 172.21.15.23:0/2709152314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48101b00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 --2- 172.21.15.23:0/2709152314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 0x7f6a481008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 -- 172.21.15.23:0/2709152314 >> 172.21.15.23:0/2709152314 conn(0x7f6a480fba20 msgr2=0x7f6a480fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 -- 172.21.15.23:0/2709152314 shutdown_connections 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.532+0000 7f6a50223640 1 -- 172.21.15.23:0/2709152314 wait complete. 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 Processor -- start 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 -- start start 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 0x7f6a4810ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4810a030 con 0x7f6a48101680 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4810a1a0 con 0x7f6a48100490 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a50223640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481a1610 con 0x7f6a48102040 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47948/0 (socket says 172.21.15.23:47948) 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 -- 172.21.15.23:0/3401998902 learned_addr learned my addr 172.21.15.23:0/3401998902 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4dfbf640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 0x7f6a4810ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4e7c0640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 -- 172.21.15.23:0/3401998902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 msgr2=0x7f6a481098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 -- 172.21.15.23:0/3401998902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 msgr2=0x7f6a4810ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 0x7f6a4810ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4d7be640 1 -- 172.21.15.23:0/3401998902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44009560 con 0x7f6a48101680 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.533+0000 7f6a4e7c0640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:35:29.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a4d7be640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6a44036040 tx=0x7f6a440040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a440044c0 con 0x7f6a48101680 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a440052d0 con 0x7f6a48101680 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481a1860 con 0x7f6a48101680 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a44003e10 con 0x7f6a48101680 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.534+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a481a1d70 con 0x7f6a48101680 2024-03-16T20:35:29.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.535+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48107800 con 0x7f6a48101680 2024-03-16T20:35:29.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.536+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6a44005440 con 0x7f6a48101680 2024-03-16T20:35:29.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.536+0000 7f6a3affd640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 0x7f6a340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.536+0000 7f6a4dfbf640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 0x7f6a340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.536+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f6a440efab0 con 0x7f6a48101680 2024-03-16T20:35:29.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.537+0000 7f6a4dfbf640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 0x7f6a340a3ef0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6a481014e0 tx=0x7f6a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:29.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.541+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6a440b8980 con 0x7f6a48101680 2024-03-16T20:35:29.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.725+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a4802cf90 con 0x7f6a48101680 2024-03-16T20:35:29.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.726+0000 7f6a3affd640 1 -- 172.21.15.23:0/3401998902 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f6a440bc040 con 0x7f6a48101680 2024-03-16T20:35:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:29.731 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:29.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 msgr2=0x7f6a340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 0x7f6a340a3ef0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6a481014e0 tx=0x7f6a3c005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 msgr2=0x7f6a48109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6a44036040 tx=0x7f6a440040c0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 shutdown_connections 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6a340a1a30 0x7f6a340a3ef0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.728+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a48102040 0x7f6a481098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.729+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a48101680 0x7f6a48109390 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.729+0000 7f6a50223640 1 --2- 172.21.15.23:0/3401998902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100490 0x7f6a4810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.729+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 >> 172.21.15.23:0/3401998902 conn(0x7f6a480fba20 msgr2=0x7f6a481049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.729+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 shutdown_connections 2024-03-16T20:35:29.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.729+0000 7f6a50223640 1 -- 172.21.15.23:0/3401998902 wait complete. 2024-03-16T20:35:29.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:35:29.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:35:29.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 -- 172.21.15.23:0/2243176910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c102820 msgr2=0x7ff38c102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 --2- 172.21.15.23:0/2243176910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c102820 0x7ff38c102ca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009900 tx=0x7ff37c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 -- 172.21.15.23:0/2243176910 shutdown_connections 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 --2- 172.21.15.23:0/2243176910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c1031e0 0x7ff38c109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 --2- 172.21.15.23:0/2243176910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c102820 0x7ff38c102ca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 --2- 172.21.15.23:0/2243176910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 -- 172.21.15.23:0/2243176910 >> 172.21.15.23:0/2243176910 conn(0x7ff38c0fb0e0 msgr2=0x7ff38c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 -- 172.21.15.23:0/2243176910 shutdown_connections 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.871+0000 7ff393e6b640 1 -- 172.21.15.23:0/2243176910 wait complete. 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 Processor -- start 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 -- start start 2024-03-16T20:35:29.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c102820 0x7ff38c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 0x7ff38c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c19f080 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c19f1f0 con 0x7ff38c1031e0 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff393e6b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c19f3b0 con 0x7ff38c102820 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff391c07640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff391c07640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47968/0 (socket says 172.21.15.23:47968) 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff391c07640 1 -- 172.21.15.23:0/3337604237 learned_addr learned my addr 172.21.15.23:0/3337604237 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff392408640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 0x7ff38c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff391406640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c102820 0x7ff38c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.872+0000 7ff391c07640 1 -- 172.21.15.23:0/3337604237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c102820 msgr2=0x7ff38c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff391c07640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c102820 0x7ff38c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff391c07640 1 -- 172.21.15.23:0/3337604237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 msgr2=0x7ff38c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff391c07640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 0x7ff38c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff391c07640 1 -- 172.21.15.23:0/3337604237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388009490 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff392408640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 0x7ff38c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff391c07640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9c0 tx=0x7ff38800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38800cde0 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c009560 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff388010910 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388018a10 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.873+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c19f9e0 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.874+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c108950 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.875+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff388010a80 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.876+0000 7ff382ffd640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 0x7ff3780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.876+0000 7ff391406640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 0x7ff3780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.876+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff388014070 con 0x7ff38c0ff7b0 2024-03-16T20:35:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.876+0000 7ff391406640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 0x7ff3780a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009230 tx=0x7ff37c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:29.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:29.880+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff3880d3020 con 0x7ff38c0ff7b0 2024-03-16T20:35:30.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.065+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff38c02d000 con 0x7ff38c0ff7b0 2024-03-16T20:35:30.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.065+0000 7ff382ffd640 1 -- 172.21.15.23:0/3337604237 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff388096ce0 con 0x7ff38c0ff7b0 2024-03-16T20:35:30.070 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:35:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 msgr2=0x7ff3780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 0x7ff3780a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009230 tx=0x7ff37c005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 msgr2=0x7ff38c197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff38800e9c0 tx=0x7ff38800ee80 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 shutdown_connections 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff3780a1a30 0x7ff3780a3ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff38c1031e0 0x7ff38c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff38c102820 0x7ff38c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 --2- 172.21.15.23:0/3337604237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38c0ff7b0 0x7ff38c197590 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 >> 172.21.15.23:0/3337604237 conn(0x7ff38c0fb0e0 msgr2=0x7ff38c105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 shutdown_connections 2024-03-16T20:35:30.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.067+0000 7ff393e6b640 1 -- 172.21.15.23:0/3337604237 wait complete. 2024-03-16T20:35:30.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:35:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/1890218418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc101520 msgr2=0x7ff4bc101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/1890218418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc101520 0x7ff4bc101940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009900 tx=0x7ff4b002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/1890218418 shutdown_connections 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/1890218418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/1890218418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 0x7ff4bc102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/1890218418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc101520 0x7ff4bc101940 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/1890218418 >> 172.21.15.23:0/1890218418 conn(0x7ff4bc0fca90 msgr2=0x7ff4bc0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/1890218418 shutdown_connections 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.186+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/1890218418 wait complete. 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 Processor -- start 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 -- start start 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 0x7ff4bc197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 0x7ff4bc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19f0f0 con 0x7ff4bc1030d0 2024-03-16T20:35:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19f260 con 0x7ff4bc101520 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c2aeb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19f420 con 0x7ff4bc102710 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c0887640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 0x7ff4bc197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47986/0 (socket says 172.21.15.23:47986) 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 -- 172.21.15.23:0/897210425 learned_addr learned my addr 172.21.15.23:0/897210425 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4bbfff640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 0x7ff4bc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 -- 172.21.15.23:0/897210425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 msgr2=0x7ff4bc197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 0x7ff4bc197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 -- 172.21.15.23:0/897210425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 msgr2=0x7ff4bc197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 0x7ff4bc197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 -- 172.21.15.23:0/897210425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4009840 con 0x7ff4bc1030d0 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c0887640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 0x7ff4bc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.187+0000 7ff4c1088640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400efd0 tx=0x7ff4b400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.188+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4009b40 con 0x7ff4bc1030d0 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.188+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b4003b70 con 0x7ff4bc1030d0 2024-03-16T20:35:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.188+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b0009560 con 0x7ff4bc1030d0 2024-03-16T20:35:30.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.188+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b40108f0 con 0x7ff4bc1030d0 2024-03-16T20:35:30.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.188+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc19fa30 con 0x7ff4bc1030d0 2024-03-16T20:35:30.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.189+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc108930 con 0x7ff4bc1030d0 2024-03-16T20:35:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.190+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff4b4003ce0 con 0x7ff4bc1030d0 2024-03-16T20:35:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.191+0000 7ff4b9ffb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 0x7ff4ac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.191+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff4b4014070 con 0x7ff4bc1030d0 2024-03-16T20:35:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.191+0000 7ff4c0887640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 0x7ff4ac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.192+0000 7ff4c0887640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 0x7ff4ac0a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005ab0 tx=0x7ff4b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.196+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff4b4096bd0 con 0x7ff4bc1030d0 2024-03-16T20:35:30.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:30 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3401998902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:30.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:30 smithi023 ceph-mon[31879]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T20:35:30.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.379+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4bc02d000 con 0x7ff4bc1030d0 2024-03-16T20:35:30.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.380+0000 7ff4b9ffb640 1 -- 172.21.15.23:0/897210425 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4b409a290 con 0x7ff4bc1030d0 2024-03-16T20:35:30.385 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 msgr2=0x7ff4ac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 0x7ff4ac0a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005ab0 tx=0x7ff4b0005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 msgr2=0x7ff4bc19ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400efd0 tx=0x7ff4b400c6a0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 shutdown_connections 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4ac0a1a30 0x7ff4ac0a3ef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1030d0 0x7ff4bc19ebb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4bc102710 0x7ff4bc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 --2- 172.21.15.23:0/897210425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4bc101520 0x7ff4bc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 >> 172.21.15.23:0/897210425 conn(0x7ff4bc0fca90 msgr2=0x7ff4bc105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 shutdown_connections 2024-03-16T20:35:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.382+0000 7ff4c2aeb640 1 -- 172.21.15.23:0/897210425 wait complete. 2024-03-16T20:35:30.398 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:35:30.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:30 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3401998902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:30.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:30 smithi031 ceph-mon[36281]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T20:35:30.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:30 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3401998902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:30.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:30 smithi189 ceph-mon[36655]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- 172.21.15.23:0/420609160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d41015f0 msgr2=0x7f66d4101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 --2- 172.21.15.23:0/420609160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d41015f0 0x7f66d4101a70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009900 tx=0x7f66d002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- 172.21.15.23:0/420609160 shutdown_connections 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 --2- 172.21.15.23:0/420609160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4101fb0 0x7f66d4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 --2- 172.21.15.23:0/420609160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d41015f0 0x7f66d4101a70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 --2- 172.21.15.23:0/420609160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d4100400 0x7f66d4100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- 172.21.15.23:0/420609160 >> 172.21.15.23:0/420609160 conn(0x7f66d40fb990 msgr2=0x7f66d40fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- 172.21.15.23:0/420609160 shutdown_connections 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- 172.21.15.23:0/420609160 wait complete. 2024-03-16T20:35:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 Processor -- start 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.501+0000 7f66db691640 1 -- start start 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d41015f0 0x7f66d419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d419f150 con 0x7f66d4101fb0 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d419f2c0 con 0x7f66d4100400 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d419f480 con 0x7f66d41015f0 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d942d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d942d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:40660/0 (socket says 172.21.15.23:40660) 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48006/0 (socket says 172.21.15.23:48006) 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d942d640 1 -- 172.21.15.23:0/455147372 learned_addr learned my addr 172.21.15.23:0/455147372 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d8c2c640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d41015f0 0x7f66d419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 -- 172.21.15.23:0/455147372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d41015f0 msgr2=0x7f66d419c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d41015f0 0x7f66d419c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 -- 172.21.15.23:0/455147372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 msgr2=0x7f66d406a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 -- 172.21.15.23:0/455147372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc009840 con 0x7f66d4101fb0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d942d640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66d9c2e640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00ec50 tx=0x7f66cc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc003b70 con 0x7f66d4101fb0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66cc00ce70 con 0x7f66d4101fb0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d0009560 con 0x7f66d4101fb0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.502+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc010830 con 0x7f66d4101fb0 2024-03-16T20:35:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.503+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d419fa00 con 0x7f66d4101fb0 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.504+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d4068730 con 0x7f66d4101fb0 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.504+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f66cc004020 con 0x7f66d4101fb0 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.505+0000 7f66ca7fc640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 0x7f66c00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.505+0000 7f66d942d640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 0x7f66c00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.505+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f66cc014070 con 0x7f66d4101fb0 2024-03-16T20:35:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.506+0000 7f66d942d640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 0x7f66c00a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f66c40058c0 tx=0x7f66c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.510+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f66cc096d20 con 0x7f66d4101fb0 2024-03-16T20:35:30.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.693+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66d402cf60 con 0x7f66d4101fb0 2024-03-16T20:35:30.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.693+0000 7f66ca7fc640 1 -- 172.21.15.23:0/455147372 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66cc003e60 con 0x7f66d4101fb0 2024-03-16T20:35:30.698 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:35:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 msgr2=0x7f66c00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 0x7f66c00a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f66c40058c0 tx=0x7f66c4005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 msgr2=0x7f66d419e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00ec50 tx=0x7f66cc00c6a0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 shutdown_connections 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f66c00a1a30 0x7f66c00a3ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66d4101fb0 0x7f66d419e970 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66d41015f0 0x7f66d419c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 --2- 172.21.15.23:0/455147372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d4100400 0x7f66d406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 >> 172.21.15.23:0/455147372 conn(0x7f66d40fb990 msgr2=0x7f66d4104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 shutdown_connections 2024-03-16T20:35:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.697+0000 7f66db691640 1 -- 172.21.15.23:0/455147372 wait complete. 2024-03-16T20:35:30.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 -- 172.21.15.23:0/1964216696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 msgr2=0x7f0d10106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 --2- 172.21.15.23:0/1964216696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10106cf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009900 tx=0x7f0d0002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 -- 172.21.15.23:0/1964216696 shutdown_connections 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 --2- 172.21.15.23:0/1964216696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10107230 0x7f0d10109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 --2- 172.21.15.23:0/1964216696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10106cf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 --2- 172.21.15.23:0/1964216696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10101fd0 0x7f0d101043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 -- 172.21.15.23:0/1964216696 >> 172.21.15.23:0/1964216696 conn(0x7f0d100fb980 msgr2=0x7f0d100fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:30.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 -- 172.21.15.23:0/1964216696 shutdown_connections 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.819+0000 7f0d1856c640 1 -- 172.21.15.23:0/1964216696 wait complete. 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 Processor -- start 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 -- start start 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 0x7f0d101975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10107230 0x7f0d1019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1019f0c0 con 0x7f0d10104900 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1019f230 con 0x7f0d10101fd0 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d1856c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1019f3f0 con 0x7f0d10107230 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d16308640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 0x7f0d101975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41566/0 (socket says 172.21.15.23:41566) 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 -- 172.21.15.23:0/3476935907 learned_addr learned my addr 172.21.15.23:0/3476935907 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d16308640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 0x7f0d101975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39258/0 (socket says 172.21.15.23:39258) 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d16b09640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10107230 0x7f0d1019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 -- 172.21.15.23:0/3476935907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10107230 msgr2=0x7f0d1019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10107230 0x7f0d1019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 -- 172.21.15.23:0/3476935907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 msgr2=0x7f0d101975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 0x7f0d101975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.820+0000 7f0d15b07640 1 -- 172.21.15.23:0/3476935907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c009490 con 0x7f0d10104900 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d15b07640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0d00005ab0 tx=0x7f0d00005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d000043d0 con 0x7f0d10104900 2024-03-16T20:35:30.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d000052d0 con 0x7f0d10104900 2024-03-16T20:35:30.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d00009560 con 0x7f0d10104900 2024-03-16T20:35:30.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00003d40 con 0x7f0d10104900 2024-03-16T20:35:30.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.821+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1019f9f0 con 0x7f0d10104900 2024-03-16T20:35:30.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.822+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d100fd470 con 0x7f0d10104900 2024-03-16T20:35:30.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.823+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0d00005440 con 0x7f0d10104900 2024-03-16T20:35:30.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.823+0000 7f0d077fe640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 0x7f0cfc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:30.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.824+0000 7f0d16308640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 0x7f0cfc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:30.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.824+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0d000efe10 con 0x7f0d10104900 2024-03-16T20:35:30.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.824+0000 7f0d16308640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 0x7f0cfc0a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009d60 tx=0x7f0d0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:30.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:30.827+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0d000b8ce0 con 0x7f0d10104900 2024-03-16T20:35:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.009+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0d1010d150 con 0x7f0d10104900 2024-03-16T20:35:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.009+0000 7f0d077fe640 1 -- 172.21.15.23:0/3476935907 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0d000430a0 con 0x7f0d10104900 2024-03-16T20:35:31.015 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 msgr2=0x7f0cfc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 0x7f0cfc0a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009d60 tx=0x7f0d0c005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 msgr2=0x7f0d10197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0d00005ab0 tx=0x7f0d00005e50 comp rx=0 tx=0).stop 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 shutdown_connections 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0cfc0a1a30 0x7f0cfc0a3ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d10107230 0x7f0d1019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d10104900 0x7f0d10197b10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 --2- 172.21.15.23:0/3476935907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d10101fd0 0x7f0d101975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.011+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 >> 172.21.15.23:0/3476935907 conn(0x7f0d100fb980 msgr2=0x7f0d10102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.012+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 shutdown_connections 2024-03-16T20:35:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.012+0000 7f0d1856c640 1 -- 172.21.15.23:0/3476935907 wait complete. 2024-03-16T20:35:31.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 -- 172.21.15.23:0/4201655476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 msgr2=0x7f7030104400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 --2- 172.21.15.23:0/4201655476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030104400 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7024009900 tx=0x7f702402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 -- 172.21.15.23:0/4201655476 shutdown_connections 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 --2- 172.21.15.23:0/4201655476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030107270 0x7f7030109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 --2- 172.21.15.23:0/4201655476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030104940 0x7f7030106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 --2- 172.21.15.23:0/4201655476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030104400 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.126+0000 7f7035bc9640 1 -- 172.21.15.23:0/4201655476 >> 172.21.15.23:0/4201655476 conn(0x7f70300fb980 msgr2=0x7f70300fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- 172.21.15.23:0/4201655476 shutdown_connections 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- 172.21.15.23:0/4201655476 wait complete. 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 Processor -- start 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- start start 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030104940 0x7f70300fffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030107270 0x7f7030100500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7030100c60 con 0x7f7030102010 2024-03-16T20:35:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7030100dd0 con 0x7f7030107270 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.127+0000 7f7035bc9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70301a1640 con 0x7f7030104940 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41584/0 (socket says 172.21.15.23:41584) 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 -- 172.21.15.23:0/1349139071 learned_addr learned my addr 172.21.15.23:0/1349139071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702ffff640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030107270 0x7f7030100500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702effd640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030104940 0x7f70300fffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 -- 172.21.15.23:0/1349139071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030104940 msgr2=0x7f70300fffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030104940 0x7f70300fffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 -- 172.21.15.23:0/1349139071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030107270 msgr2=0x7f7030100500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030107270 0x7f7030100500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 -- 172.21.15.23:0/1349139071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020009490 con 0x7f7030102010 2024-03-16T20:35:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702f7fe640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7024036040 tx=0x7f7024005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70240043d0 con 0x7f7030102010 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70240052d0 con 0x7f7030102010 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7028009840 con 0x7f7030102010 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7024003bc0 con 0x7f7030102010 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.128+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7024009560 con 0x7f7030102010 2024-03-16T20:35:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.129+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70300fd470 con 0x7f7030102010 2024-03-16T20:35:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.130+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7024005440 con 0x7f7030102010 2024-03-16T20:35:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.131+0000 7f702cff9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 0x7f70180a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.131+0000 7f702effd640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 0x7f70180a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.131+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f70240efef0 con 0x7f7030102010 2024-03-16T20:35:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.132+0000 7f702effd640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 0x7f70180a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f70301016f0 tx=0x7f7020005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.135+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f70240b8d60 con 0x7f7030102010 2024-03-16T20:35:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:31 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3337604237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:31 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/897210425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:31 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/455147372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:31 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3476935907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.360+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f703010cf20 con 0x7f7030102010 2024-03-16T20:35:31.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.361+0000 7f702cff9640 1 -- 172.21.15.23:0/1349139071 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7f70240bc420 con 0x7f7030102010 2024-03-16T20:35:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:31.366 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:35:31.366 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 msgr2=0x7f70180a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 0x7f70180a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f70301016f0 tx=0x7f7020005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 msgr2=0x7f7030101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7024036040 tx=0x7f7024005e80 comp rx=0 tx=0).stop 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 shutdown_connections 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f70180a1ab0 0x7f70180a3f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7030107270 0x7f7030100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7030104940 0x7f70300fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 --2- 172.21.15.23:0/1349139071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7030102010 0x7f7030101980 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 >> 172.21.15.23:0/1349139071 conn(0x7f70300fb980 msgr2=0x7f7030102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 shutdown_connections 2024-03-16T20:35:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.363+0000 7f7035bc9640 1 -- 172.21.15.23:0/1349139071 wait complete. 2024-03-16T20:35:31.379 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:31.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:31 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3337604237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:31 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/897210425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:31 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/455147372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:31 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3476935907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:31 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3337604237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:31 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/897210425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:31 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/455147372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:31 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3476935907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 -- 172.21.15.23:0/1657088794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff040102700 msgr2=0x7ff040102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 --2- 172.21.15.23:0/1657088794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff040102700 0x7ff040102b80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009900 tx=0x7ff02c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 -- 172.21.15.23:0/1657088794 shutdown_connections 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 --2- 172.21.15.23:0/1657088794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0401030c0 0x7ff040109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 --2- 172.21.15.23:0/1657088794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff040102700 0x7ff040102b80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 --2- 172.21.15.23:0/1657088794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff040101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 -- 172.21.15.23:0/1657088794 >> 172.21.15.23:0/1657088794 conn(0x7ff0400fcaa0 msgr2=0x7ff0400feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 -- 172.21.15.23:0/1657088794 shutdown_connections 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 -- 172.21.15.23:0/1657088794 wait complete. 2024-03-16T20:35:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.477+0000 7ff04709a640 1 Processor -- start 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 -- start start 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff0401975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff040102700 0x7ff040197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04019f090 con 0x7ff0401030c0 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04019f200 con 0x7ff040101510 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff04709a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04019f3c0 con 0x7ff040102700 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff044e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff0401975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41594/0 (socket says 172.21.15.23:41594) 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 -- 172.21.15.23:0/3524850492 learned_addr learned my addr 172.21.15.23:0/3524850492 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff03ffff640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff040102700 0x7ff040197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 -- 172.21.15.23:0/3524850492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff040102700 msgr2=0x7ff040197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff040102700 0x7ff040197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 -- 172.21.15.23:0/3524850492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 msgr2=0x7ff0401975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff0401975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 -- 172.21.15.23:0/3524850492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff034009490 con 0x7ff0401030c0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff044e36640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff0401975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff045637640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff03800eb10 tx=0x7ff03800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.478+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038004b20 con 0x7ff0401030c0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.479+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02c009560 con 0x7ff0401030c0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.479+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff038004c80 con 0x7ff0401030c0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.479+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038010670 con 0x7ff0401030c0 2024-03-16T20:35:31.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.479+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04019fa40 con 0x7ff0401030c0 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.480+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff040108830 con 0x7ff0401030c0 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.481+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff038005020 con 0x7ff0401030c0 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.481+0000 7ff03dffb640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 0x7ff0300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.481+0000 7ff044e36640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 0x7ff0300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.481+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff038014070 con 0x7ff0401030c0 2024-03-16T20:35:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.482+0000 7ff044e36640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 0x7ff0300a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff040102560 tx=0x7ff034005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.485+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff038096b50 con 0x7ff0401030c0 2024-03-16T20:35:31.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.668+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff04002d000 con 0x7ff0401030c0 2024-03-16T20:35:31.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.670+0000 7ff03dffb640 1 -- 172.21.15.23:0/3524850492 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7ff03809a210 con 0x7ff0401030c0 2024-03-16T20:35:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:31.675 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 msgr2=0x7ff0300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 0x7ff0300a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff040102560 tx=0x7ff034005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 msgr2=0x7ff04019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff03800eb10 tx=0x7ff03800c2d0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 shutdown_connections 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0300a1a30 0x7ff0300a3ef0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0401030c0 0x7ff04019eb50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff040102700 0x7ff040197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 --2- 172.21.15.23:0/3524850492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff040101510 0x7ff0401975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 >> 172.21.15.23:0/3524850492 conn(0x7ff0400fcaa0 msgr2=0x7ff040105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 shutdown_connections 2024-03-16T20:35:31.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.672+0000 7ff04709a640 1 -- 172.21.15.23:0/3524850492 wait complete. 2024-03-16T20:35:31.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:35:31.688 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- 172.21.15.23:0/1039397115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c1026f0 msgr2=0x7f0e3c102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 --2- 172.21.15.23:0/1039397115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c102b70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009860 tx=0x7f0e2c02bb70 comp rx=0 tx=0).stop 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- 172.21.15.23:0/1039397115 shutdown_connections 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 --2- 172.21.15.23:0/1039397115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 --2- 172.21.15.23:0/1039397115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c102b70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 --2- 172.21.15.23:0/1039397115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c101500 0x7f0e3c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- 172.21.15.23:0/1039397115 >> 172.21.15.23:0/1039397115 conn(0x7f0e3c0fca90 msgr2=0x7f0e3c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- 172.21.15.23:0/1039397115 shutdown_connections 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- 172.21.15.23:0/1039397115 wait complete. 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 Processor -- start 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.789+0000 7f0e44084640 1 -- start start 2024-03-16T20:35:31.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c101500 0x7f0e3c1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19f0a0 con 0x7f0e3c1030b0 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19f210 con 0x7f0e3c101500 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19f3d0 con 0x7f0e3c1026f0 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41612/0 (socket says 172.21.15.23:41612) 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 -- 172.21.15.23:0/3032482667 learned_addr learned my addr 172.21.15.23:0/3032482667 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e41e20640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c101500 0x7f0e3c1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e4161f640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 -- 172.21.15.23:0/3032482667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c1026f0 msgr2=0x7f0e3c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 -- 172.21.15.23:0/3032482667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c101500 msgr2=0x7f0e3c1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c101500 0x7f0e3c1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 -- 172.21.15.23:0/3032482667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e34009900 con 0x7f0e3c1030b0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e42621640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400efc0 tx=0x7f0e3400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34003970 con 0x7f0e3c1030b0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c009490 con 0x7f0e3c1030b0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e34004020 con 0x7f0e3c1030b0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3400b9b0 con 0x7f0e3c1030b0 2024-03-16T20:35:31.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.790+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c19f9b0 con 0x7f0e3c1030b0 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.791+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c1088c0 con 0x7f0e3c1030b0 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.792+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0e3400a040 con 0x7f0e3c1030b0 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.793+0000 7f0e32ffd640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 0x7f0e280a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.793+0000 7f0e41e20640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 0x7f0e280a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.793+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0e340cd750 con 0x7f0e3c1030b0 2024-03-16T20:35:31.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.793+0000 7f0e41e20640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 0x7f0e280a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c102550 tx=0x7f0e38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:31.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.797+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0e34096620 con 0x7f0e3c1030b0 2024-03-16T20:35:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.980+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e3c02d000 con 0x7f0e3c1030b0 2024-03-16T20:35:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.980+0000 7f0e32ffd640 1 -- 172.21.15.23:0/3032482667 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e34004190 con 0x7f0e3c1030b0 2024-03-16T20:35:31.985 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:35:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 msgr2=0x7f0e280a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 0x7f0e280a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c102550 tx=0x7f0e38005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 msgr2=0x7f0e3c19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400efc0 tx=0x7f0e3400c800 comp rx=0 tx=0).stop 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 shutdown_connections 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0e280a1a30 0x7f0e280a3ef0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e3c1030b0 0x7f0e3c19eb60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e3c1026f0 0x7f0e3c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 --2- 172.21.15.23:0/3032482667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e3c101500 0x7f0e3c1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 >> 172.21.15.23:0/3032482667 conn(0x7f0e3c0fca90 msgr2=0x7f0e3c105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 shutdown_connections 2024-03-16T20:35:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:31.982+0000 7f0e44084640 1 -- 172.21.15.23:0/3032482667 wait complete. 2024-03-16T20:35:31.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 -- 172.21.15.23:0/2818918759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 msgr2=0x7f3cbc109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/2818918759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc109620 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009ca0 tx=0x7f3cb802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 -- 172.21.15.23:0/2818918759 shutdown_connections 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/2818918759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc109620 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/2818918759 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/2818918759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 0x7f3cbc1043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.100+0000 7f3cc36b5640 1 -- 172.21.15.23:0/2818918759 >> 172.21.15.23:0/2818918759 conn(0x7f3cbc0fb980 msgr2=0x7f3cbc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- 172.21.15.23:0/2818918759 shutdown_connections 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- 172.21.15.23:0/2818918759 wait complete. 2024-03-16T20:35:32.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 Processor -- start 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- start start 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 0x7f3cbc101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc100c20 con 0x7f3cbc107230 2024-03-16T20:35:32.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc100d90 con 0x7f3cbc101fd0 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc36b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc1a15d0 con 0x7f3cbc104900 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc1c52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc1c52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41624/0 (socket says 172.21.15.23:41624) 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc1c52640 1 -- 172.21.15.23:0/1341334566 learned_addr learned my addr 172.21.15.23:0/1341334566 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc0c50640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.101+0000 7f3cc1451640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 0x7f3cbc101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 -- 172.21.15.23:0/1341334566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 msgr2=0x7f3cbc0fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc0fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 -- 172.21.15.23:0/1341334566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 msgr2=0x7f3cbc101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 0x7f3cbc101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 -- 172.21.15.23:0/1341334566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8009900 con 0x7f3cbc107230 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc0c50640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc0fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc1c52640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8033000 tx=0x7f3cb8009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb80329b0 con 0x7f3cbc107230 2024-03-16T20:35:32.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cb8003e50 con 0x7f3cbc107230 2024-03-16T20:35:32.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc1a1820 con 0x7f3cbc107230 2024-03-16T20:35:32.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8040450 con 0x7f3cbc107230 2024-03-16T20:35:32.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.102+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc1a1d20 con 0x7f3cbc107230 2024-03-16T20:35:32.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.103+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc0fd470 con 0x7f3cbc107230 2024-03-16T20:35:32.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.104+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3cb800a040 con 0x7f3cbc107230 2024-03-16T20:35:32.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.105+0000 7f3cb27fc640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 0x7f3ca80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.105+0000 7f3cc1451640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 0x7f3ca80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.105+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3cb80f0450 con 0x7f3cbc107230 2024-03-16T20:35:32.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.105+0000 7f3cc1451640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 0x7f3ca80a3f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009bd0 tx=0x7f3cb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.109+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3cb80b9320 con 0x7f3cbc107230 2024-03-16T20:35:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.290+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3cbc02d000 con 0x7f3cbc107230 2024-03-16T20:35:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.291+0000 7f3cb27fc640 1 -- 172.21.15.23:0/1341334566 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3cb80bc9e0 con 0x7f3cbc107230 2024-03-16T20:35:32.295 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.292+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 msgr2=0x7f3ca80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.292+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 0x7f3ca80a3f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009bd0 tx=0x7f3cb4005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.292+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 msgr2=0x7f3cbc1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.292+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8033000 tx=0x7f3cb8009360 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 shutdown_connections 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3ca80a1ab0 0x7f3ca80a3f70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cbc107230 0x7f3cbc1004c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cbc104900 0x7f3cbc0fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 --2- 172.21.15.23:0/1341334566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3cbc101fd0 0x7f3cbc101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 >> 172.21.15.23:0/1341334566 conn(0x7f3cbc0fb980 msgr2=0x7f3cbc102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 shutdown_connections 2024-03-16T20:35:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.293+0000 7f3cc36b5640 1 -- 172.21.15.23:0/1341334566 wait complete. 2024-03-16T20:35:32.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:32 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1349139071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:32 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3524850492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:32 smithi023 ceph-mon[31879]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T20:35:32.307 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:32 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3032482667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:32.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:35:32.339 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:32 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1349139071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.339 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:32 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3524850492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.339 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:32 smithi031 ceph-mon[36281]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T20:35:32.340 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:32 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3032482667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:32.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:32 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1349139071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:32 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3524850492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:32.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:32 smithi189 ceph-mon[36655]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-16T20:35:32.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:32 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3032482667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3433540391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac0ff7e0 msgr2=0x7f78ac0ffc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3433540391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac0ffc00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f789c009900 tx=0x7f789c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3433540391 shutdown_connections 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3433540391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3433540391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac1027b0 0x7f78ac102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3433540391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac0ffc00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3433540391 >> 172.21.15.23:0/3433540391 conn(0x7f78ac0fb0f0 msgr2=0x7f78ac0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3433540391 shutdown_connections 2024-03-16T20:35:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.409+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3433540391 wait complete. 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 Processor -- start 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 -- start start 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac19f0c0 con 0x7f78ac1027b0 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac19f230 con 0x7f78ac103170 2024-03-16T20:35:32.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78b0b5e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac19f3f0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78a9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78a9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41646/0 (socket says 172.21.15.23:41646) 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78a9d9b640 1 -- 172.21.15.23:0/3068486956 learned_addr learned my addr 172.21.15.23:0/3068486956 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aad9d640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 -- 172.21.15.23:0/3068486956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 msgr2=0x7f78ac19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 -- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 msgr2=0x7f78ac197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aa59c640 1 -- 172.21.15.23:0/3068486956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c009560 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78a9d9b640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.410+0000 7f78aad9d640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f78aa59c640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac1975d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f789c036040 tx=0x7f789c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c004aa0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac19f670 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f789c0052d0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789c003df0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.411+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac19fba0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.412+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac108900 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.413+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f789c005440 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.413+0000 7f789b7fe640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 0x7f78940a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.413+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f789c0efd00 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.413+0000 7f78a9d9b640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 0x7f78940a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.414+0000 7f78a9d9b640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 0x7f78940a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f78ac198ae0 tx=0x7f78a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.417+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f789c0b8bb0 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.601+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f78ac02cf90 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.601+0000 7f789b7fe640 1 -- 172.21.15.23:0/3068486956 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f789c0bc270 con 0x7f78ac0ff7e0 2024-03-16T20:35:32.606 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 msgr2=0x7f78940a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 0x7f78940a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f78ac198ae0 tx=0x7f78a0005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 msgr2=0x7f78ac1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac1975d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f789c036040 tx=0x7f789c004270 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 shutdown_connections 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78940a1a30 0x7f78940a3ef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78ac103170 0x7f78ac19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78ac1027b0 0x7f78ac197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 --2- 172.21.15.23:0/3068486956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78ac0ff7e0 0x7f78ac1975d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 >> 172.21.15.23:0/3068486956 conn(0x7f78ac0fb0f0 msgr2=0x7f78ac105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.603+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 shutdown_connections 2024-03-16T20:35:32.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.604+0000 7f78b0b5e640 1 -- 172.21.15.23:0/3068486956 wait complete. 2024-03-16T20:35:32.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:35:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 -- 172.21.15.23:0/3798208160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 msgr2=0x7f5be8101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/3798208160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be8101920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009900 tx=0x7f5be402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 -- 172.21.15.23:0/3798208160 shutdown_connections 2024-03-16T20:35:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/3798208160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 0x7f5be8109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/3798208160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 0x7f5be8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/3798208160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be8101920 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 -- 172.21.15.23:0/3798208160 >> 172.21.15.23:0/3798208160 conn(0x7f5be80fca90 msgr2=0x7f5be80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 -- 172.21.15.23:0/3798208160 shutdown_connections 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.719+0000 7f5bef7c7640 1 -- 172.21.15.23:0/3798208160 wait complete. 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 Processor -- start 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 -- start start 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 0x7f5be8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 0x7f5be819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819f0c0 con 0x7f5be8101500 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819f230 con 0x7f5be81026f0 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bef7c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819f3f0 con 0x7f5be81030b0 2024-03-16T20:35:32.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41670/0 (socket says 172.21.15.23:41670) 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 -- 172.21.15.23:0/357608603 learned_addr learned my addr 172.21.15.23:0/357608603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5becd62640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 0x7f5be8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bedd64640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 0x7f5be819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 -- 172.21.15.23:0/357608603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 msgr2=0x7f5be819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 0x7f5be819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 -- 172.21.15.23:0/357608603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 msgr2=0x7f5be8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 0x7f5be8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.720+0000 7f5bed563640 1 -- 172.21.15.23:0/357608603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8009490 con 0x7f5be8101500 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bed563640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5be4005ab0 tx=0x7f5be4005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40043d0 con 0x7f5be8101500 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5be40052d0 con 0x7f5be8101500 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be4009560 con 0x7f5be8101500 2024-03-16T20:35:32.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4003d40 con 0x7f5be8101500 2024-03-16T20:35:32.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.721+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be819f9f0 con 0x7f5be8101500 2024-03-16T20:35:32.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.722+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be81088a0 con 0x7f5be8101500 2024-03-16T20:35:32.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.723+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5be4005440 con 0x7f5be8101500 2024-03-16T20:35:32.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.723+0000 7f5bde7fc640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 0x7f5bd40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:32.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.723+0000 7f5becd62640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 0x7f5bd40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:32.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.723+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f5be40ef860 con 0x7f5be8101500 2024-03-16T20:35:32.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.724+0000 7f5becd62640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 0x7f5bd40a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5be8198ae0 tx=0x7f5bd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:32.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.728+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5be40b8730 con 0x7f5be8101500 2024-03-16T20:35:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.909+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5be802cf90 con 0x7f5be8101500 2024-03-16T20:35:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.909+0000 7f5bde7fc640 1 -- 172.21.15.23:0/357608603 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5be40bbdf0 con 0x7f5be8101500 2024-03-16T20:35:32.914 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 msgr2=0x7f5bd40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 0x7f5bd40a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5be8198ae0 tx=0x7f5bd8005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 msgr2=0x7f5be81975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5be4005ab0 tx=0x7f5be4005e50 comp rx=0 tx=0).stop 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 shutdown_connections 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f5bd40a1a30 0x7f5bd40a3ef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5be81030b0 0x7f5be819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be81026f0 0x7f5be8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 --2- 172.21.15.23:0/357608603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5be8101500 0x7f5be81975d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 >> 172.21.15.23:0/357608603 conn(0x7f5be80fca90 msgr2=0x7f5be8105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 shutdown_connections 2024-03-16T20:35:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:32.911+0000 7f5bef7c7640 1 -- 172.21.15.23:0/357608603 wait complete. 2024-03-16T20:35:32.927 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 -- 172.21.15.23:0/1185458762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 msgr2=0x7f97681088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 --2- 172.21.15.23:0/1185458762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681088d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9760009ca0 tx=0x7f976002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 -- 172.21.15.23:0/1185458762 shutdown_connections 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 --2- 172.21.15.23:0/1185458762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681088d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 --2- 172.21.15.23:0/1185458762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 0x7f9768101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 --2- 172.21.15.23:0/1185458762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f97681008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 -- 172.21.15.23:0/1185458762 >> 172.21.15.23:0/1185458762 conn(0x7f97680fba20 msgr2=0x7f97680fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 -- 172.21.15.23:0/1185458762 shutdown_connections 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.025+0000 7f976d108640 1 -- 172.21.15.23:0/1185458762 wait complete. 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 Processor -- start 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 -- start start 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f976806a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 0x7f976819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681a1280 con 0x7f9768102040 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681a13f0 con 0x7f9768100490 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976d108640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681a15b0 con 0x7f9768101680 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f9766d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f976806a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41698/0 (socket says 172.21.15.23:41698) 2024-03-16T20:35:33.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 -- 172.21.15.23:0/3313823357 learned_addr learned my addr 172.21.15.23:0/3313823357 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f975ffff640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 0x7f976819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 -- 172.21.15.23:0/3313823357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 msgr2=0x7f976819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 0x7f976819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.026+0000 7f976759e640 1 -- 172.21.15.23:0/3313823357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 msgr2=0x7f976806a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f976759e640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f976806a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f976759e640 1 -- 172.21.15.23:0/3313823357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758009490 con 0x7f9768102040 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f9766d9d640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f976806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f976759e640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9760037040 tx=0x7f9760009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97600329b0 con 0x7f9768102040 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9760003e50 con 0x7f9768102040 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9760009900 con 0x7f9768102040 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760040450 con 0x7f9768102040 2024-03-16T20:35:33.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.027+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97681a1af0 con 0x7f9768102040 2024-03-16T20:35:33.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.028+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9768107750 con 0x7f9768102040 2024-03-16T20:35:33.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.029+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f976000a040 con 0x7f9768102040 2024-03-16T20:35:33.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.030+0000 7f9764d99640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 0x7f97540a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.030+0000 7f9766d9d640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 0x7f97540a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.030+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f97600f0860 con 0x7f9768102040 2024-03-16T20:35:33.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.030+0000 7f9766d9d640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 0x7f97540a3f20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97681014e0 tx=0x7f9758005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.033+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f97600b97c0 con 0x7f9768102040 2024-03-16T20:35:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.257+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f976810c030 con 0x7f9768102040 2024-03-16T20:35:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.258+0000 7f9764d99640 1 -- 172.21.15.23:0/3313823357 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7f9760034e60 con 0x7f9768102040 2024-03-16T20:35:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:33.263 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:35:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 msgr2=0x7f97540a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 0x7f97540a3f20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97681014e0 tx=0x7f9758005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 msgr2=0x7f97681a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9760037040 tx=0x7f9760009360 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 shutdown_connections 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97540a1a60 0x7f97540a3f20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9768102040 0x7f97681a0bc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9768101680 0x7f976819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 --2- 172.21.15.23:0/3313823357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9768100490 0x7f976806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 >> 172.21.15.23:0/3313823357 conn(0x7f97680fba20 msgr2=0x7f9768104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 shutdown_connections 2024-03-16T20:35:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.260+0000 7f976d108640 1 -- 172.21.15.23:0/3313823357 wait complete. 2024-03-16T20:35:33.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:33 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1341334566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:33 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3068486956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:33 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/357608603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- 172.21.15.23:0/4284340967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40106080 msgr2=0x7f3b40108540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 --2- 172.21.15.23:0/4284340967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40106080 0x7f3b40108540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009900 tx=0x7f3b3802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- 172.21.15.23:0/4284340967 shutdown_connections 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 --2- 172.21.15.23:0/4284340967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40106080 0x7f3b40108540 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 --2- 172.21.15.23:0/4284340967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40103600 0x7f3b401059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 --2- 172.21.15.23:0/4284340967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 0x7f3b40103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- 172.21.15.23:0/4284340967 >> 172.21.15.23:0/4284340967 conn(0x7f3b400facf0 msgr2=0x7f3b400fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- 172.21.15.23:0/4284340967 shutdown_connections 2024-03-16T20:35:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- 172.21.15.23:0/4284340967 wait complete. 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 Processor -- start 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.373+0000 7f3b45484640 1 -- start start 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 0x7f3b40197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40106080 0x7f3b4019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4019f0a0 con 0x7f3b40103600 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4019f210 con 0x7f3b40100c40 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4019f3d0 con 0x7f3b40106080 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41720/0 (socket says 172.21.15.23:41720) 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 -- 172.21.15.23:0/1543123002 learned_addr learned my addr 172.21.15.23:0/1543123002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3effd640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 0x7f3b40197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3f7fe640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40106080 0x7f3b4019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 -- 172.21.15.23:0/1543123002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40106080 msgr2=0x7f3b4019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40106080 0x7f3b4019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 -- 172.21.15.23:0/1543123002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 msgr2=0x7f3b40197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 0x7f3b40197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 -- 172.21.15.23:0/1543123002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b38009560 con 0x7f3b40103600 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b3e7fc640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800e9f0 tx=0x7f3b2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b280096c0 con 0x7f3b40103600 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b28005500 con 0x7f3b40103600 2024-03-16T20:35:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28010430 con 0x7f3b40103600 2024-03-16T20:35:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4019f6b0 con 0x7f3b40103600 2024-03-16T20:35:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.374+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4019fb70 con 0x7f3b40103600 2024-03-16T20:35:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.375+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b400fc7e0 con 0x7f3b40103600 2024-03-16T20:35:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.376+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3b28005020 con 0x7f3b40103600 2024-03-16T20:35:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.377+0000 7f3b37fff640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 0x7f3b2c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.377+0000 7f3b3effd640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 0x7f3b2c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.377+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3b28014070 con 0x7f3b40103600 2024-03-16T20:35:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.378+0000 7f3b3effd640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 0x7f3b2c0a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ff7e0 tx=0x7f3b30009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.382+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3b28096950 con 0x7f3b40103600 2024-03-16T20:35:33.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:33 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1341334566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:33 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3068486956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:33 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/357608603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1341334566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3068486956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:33 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/357608603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.566+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b4010cf40 con 0x7f3b40103600 2024-03-16T20:35:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.568+0000 7f3b37fff640 1 -- 172.21.15.23:0/1543123002 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f3b2809a010 con 0x7f3b40103600 2024-03-16T20:35:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:33.573 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:33.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.570+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 msgr2=0x7f3b2c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.570+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 0x7f3b2c0a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ff7e0 tx=0x7f3b30009500 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.570+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 msgr2=0x7f3b40197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.570+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800e9f0 tx=0x7f3b2800eeb0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 shutdown_connections 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3b2c0a1a30 0x7f3b2c0a3ef0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b40106080 0x7f3b4019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b40103600 0x7f3b40197b90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 --2- 172.21.15.23:0/1543123002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b40100c40 0x7f3b40197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 >> 172.21.15.23:0/1543123002 conn(0x7f3b400facf0 msgr2=0x7f3b401015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 shutdown_connections 2024-03-16T20:35:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.571+0000 7f3b45484640 1 -- 172.21.15.23:0/1543123002 wait complete. 2024-03-16T20:35:33.587 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:35:33.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 -- 172.21.15.23:0/4147864573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc1030c0 msgr2=0x7f97fc109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 --2- 172.21.15.23:0/4147864573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc1030c0 0x7f97fc109950 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009ca0 tx=0x7f97f402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 -- 172.21.15.23:0/4147864573 shutdown_connections 2024-03-16T20:35:33.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 --2- 172.21.15.23:0/4147864573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc1030c0 0x7f97fc109950 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 --2- 172.21.15.23:0/4147864573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc102700 0x7f97fc102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 --2- 172.21.15.23:0/4147864573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 -- 172.21.15.23:0/4147864573 >> 172.21.15.23:0/4147864573 conn(0x7f97fc0fcaa0 msgr2=0x7f97fc0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 -- 172.21.15.23:0/4147864573 shutdown_connections 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.684+0000 7f9803c23640 1 -- 172.21.15.23:0/4147864573 wait complete. 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 Processor -- start 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 -- start start 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 0x7f97fc0738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 0x7f97fc073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc074580 con 0x7f97fc102700 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0746f0 con 0x7f97fc101510 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f9803c23640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1a1580 con 0x7f97fc1030c0 2024-03-16T20:35:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 0x7f97fc0738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98019bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98019bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39434/0 (socket says 172.21.15.23:39434) 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98019bf640 1 -- 172.21.15.23:0/2272437868 learned_addr learned my addr 172.21.15.23:0/2272437868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98021c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 0x7f97fc073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 -- 172.21.15.23:0/2272437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 msgr2=0x7f97fc073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 0x7f97fc073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 -- 172.21.15.23:0/2272437868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 msgr2=0x7f97fc0752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 -- 172.21.15.23:0/2272437868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8009490 con 0x7f97fc102700 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98021c0640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 0x7f97fc073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98019bf640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.685+0000 7f98011be640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 0x7f97fc0738e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00ea20 tx=0x7f97ec00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.686+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00ccc0 con 0x7f97fc102700 2024-03-16T20:35:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.686+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97ec00ce20 con 0x7f97fc102700 2024-03-16T20:35:33.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.686+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f4009900 con 0x7f97fc102700 2024-03-16T20:35:33.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.686+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec018840 con 0x7f97fc102700 2024-03-16T20:35:33.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.686+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1a1c00 con 0x7f97fc102700 2024-03-16T20:35:33.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.687+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc108830 con 0x7f97fc102700 2024-03-16T20:35:33.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.688+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f97ec0189a0 con 0x7f97fc102700 2024-03-16T20:35:33.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.688+0000 7f97f2ffd640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 0x7f97e80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:33.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.688+0000 7f98019bf640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 0x7f97e80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:33.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.689+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f97ec014070 con 0x7f97fc102700 2024-03-16T20:35:33.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.689+0000 7f98019bf640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 0x7f97e80a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97f80058c0 tx=0x7f97f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:33.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.693+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f97ec053280 con 0x7f97fc102700 2024-03-16T20:35:33.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.913+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97fc02d000 con 0x7f97fc102700 2024-03-16T20:35:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.913+0000 7f97f2ffd640 1 -- 172.21.15.23:0/2272437868 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7f97ec0968f0 con 0x7f97fc102700 2024-03-16T20:35:33.918 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:33.918 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:35:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 msgr2=0x7f97e80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 0x7f97e80a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97f80058c0 tx=0x7f97f8005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 msgr2=0x7f97fc0738e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 0x7f97fc0738e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00ea20 tx=0x7f97ec00eee0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 shutdown_connections 2024-03-16T20:35:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f97e80a1ab0 0x7f97e80a3f70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc1030c0 0x7f97fc073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97fc102700 0x7f97fc0738e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 --2- 172.21.15.23:0/2272437868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97fc101510 0x7f97fc0752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 >> 172.21.15.23:0/2272437868 conn(0x7f97fc0fcaa0 msgr2=0x7f97fc105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 shutdown_connections 2024-03-16T20:35:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:33.915+0000 7f9803c23640 1 -- 172.21.15.23:0/2272437868 wait complete. 2024-03-16T20:35:33.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T20:33:49.239658+0000', 'modified': '2024-03-16T20:34:10.946679+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24511, 'mds_1': 24497, 'mds_2': 24493, 'mds_3': 24418, 'mds_4': 24416}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14643': {'gid': 14643, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/2750734224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 2750734224}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 2750734224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14655': {'gid': 14655, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6841/4113023748', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 4113023748}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 4113023748}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24416': {'gid': 24416, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.31:6833/1919282692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 1919282692}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 1919282692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6833/1083314774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 1083314774}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 1083314774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6835/354681253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6834', 'nonce': 354681253}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 354681253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24469': {'gid': 24469, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1658699703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1658699703}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1658699703}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/3204120502', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3204120502}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3204120502}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24493': {'gid': 24493, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.31:6837/424329834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6836', 'nonce': 424329834}, {'type': 'v1', 'addr': '172.21.15.31:6837', 'nonce': 424329834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.31:6839/37663771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 37663771}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 37663771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.189:6839/2333060283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2333060283}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2333060283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24511, 'qdb_cluster': [24497, 24511, 24418, 24493, 24416]} 2024-03-16T20:35:33.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-16T20:35:33.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 -- 172.21.15.23:0/3135783518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc100410 msgr2=0x7f0efc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/3135783518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc100410 0x7f0efc100830 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009900 tx=0x7f0ef802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 -- 172.21.15.23:0/3135783518 shutdown_connections 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/3135783518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc101fc0 0x7f0efc108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/3135783518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101600 0x7f0efc101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/3135783518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc100410 0x7f0efc100830 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 -- 172.21.15.23:0/3135783518 >> 172.21.15.23:0/3135783518 conn(0x7f0efc0fb9a0 msgr2=0x7f0efc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 -- 172.21.15.23:0/3135783518 shutdown_connections 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.030+0000 7f0f03bf4640 1 -- 172.21.15.23:0/3135783518 wait complete. 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 Processor -- start 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 -- start start 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 0x7f0efc06a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 0x7f0efc19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc19f100 con 0x7f0efc101600 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc19f270 con 0x7f0efc100410 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f03bf4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc19f430 con 0x7f0efc101fc0 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41750/0 (socket says 172.21.15.23:41750) 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 -- 172.21.15.23:0/2974266814 learned_addr learned my addr 172.21.15.23:0/2974266814 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f01990640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 0x7f0efc06a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f02191640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 0x7f0efc19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 -- 172.21.15.23:0/2974266814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 msgr2=0x7f0efc19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 0x7f0efc19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 -- 172.21.15.23:0/2974266814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 msgr2=0x7f0efc06a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 0x7f0efc06a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 -- 172.21.15.23:0/2974266814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4009840 con 0x7f0efc101600 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f02191640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 0x7f0efc19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f01990640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 0x7f0efc06a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.031+0000 7f0f0118f640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00e9f0 tx=0x7f0eec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.032+0000 7f0ef2ffd640 1 -- 172.21.15.23:0/2974266814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec009720 con 0x7f0efc101600 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.032+0000 7f0ef2ffd640 1 -- 172.21.15.23:0/2974266814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0eec00cd80 con 0x7f0efc101600 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.032+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef8009560 con 0x7f0efc101600 2024-03-16T20:35:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.032+0000 7f0ef2ffd640 1 -- 172.21.15.23:0/2974266814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec010690 con 0x7f0efc101600 2024-03-16T20:35:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.032+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc19fa40 con 0x7f0efc101600 2024-03-16T20:35:34.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0f067ce740 1 Processor -- start 2024-03-16T20:35:34.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0f067ce740 1 -- start start 2024-03-16T20:35:34.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0ef2ffd640 1 -- 172.21.15.23:0/2974266814 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0eec0046c0 con 0x7f0efc101600 2024-03-16T20:35:34.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.034+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f067ce740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d21aacd0 con 0x5559d21eef80 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f067ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d215df20 con 0x5559d21f4ae0 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f067ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d215b450 con 0x5559d21ef970 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f02191640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39462/0 (socket says 172.21.15.23:39462) 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f02191640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41754/0 (socket says 172.21.15.23:41754) 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f01990640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 -- 172.21.15.23:0/3509319443 learned_addr learned my addr 172.21.15.23:0/3509319443 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 -- 172.21.15.23:0/3509319443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 msgr2=0x5559d21f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 -- 172.21.15.23:0/3509319443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 msgr2=0x5559d21ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0ef2ffd640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 0x7f0ee80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 -- 172.21.15.23:0/3509319443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d210bf20 con 0x5559d21f4ae0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f02191640 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f01990640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 0x7f0ee80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0ef2ffd640 1 -- 172.21.15.23:0/2974266814 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0eec014070 con 0x7f0efc101600 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0f0118f640 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0eec09b680 tx=0x7f0eec096a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46acf712f097f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.035+0000 7f0ee7fff640 1 -- 172.21.15.23:0/3509319443 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec09a480 con 0x5559d21f4ae0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0ee7fff640 1 -- 172.21.15.23:0/3509319443 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0eec09a1f0 con 0x5559d21f4ae0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0ee7fff640 1 -- 172.21.15.23:0/3509319443 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0d3e50 con 0x5559d21f4ae0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f01990640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 0x7f0ee80a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005ab0 tx=0x7f0ef8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 -- 172.21.15.23:0/3509319443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 msgr2=0x5559d21f6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0eec09b680 tx=0x7f0eec096a10 comp rx=0 tx=0).stop 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 -- 172.21.15.23:0/3509319443 shutdown_connections 2024-03-16T20:35:34.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d21f6f70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3509319443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 -- 172.21.15.23:0/3509319443 >> 172.21.15.23:0/3509319443 conn(0x5559d21ea4f0 msgr2=0x5559d21ea8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 -- 172.21.15.23:0/3509319443 shutdown_connections 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.036+0000 7f0f067ce740 1 -- 172.21.15.23:0/3509319443 wait complete. 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 -- start start 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 Processor -- start 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d22006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d2156c30 con 0x5559d21eef80 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d20fbd30 con 0x5559d21f4ae0 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d1e2a310 con 0x5559d21ef970 2024-03-16T20:35:34.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f067ce740 10 client.0 _ll_register_callbacks cb 0x5559d2103d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f01990640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f01990640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41762/0 (socket says 172.21.15.23:41762) 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f01990640 1 -- 172.21.15.23:0/1325545795 learned_addr learned my addr 172.21.15.23:0/1325545795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f02191640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d22006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.037+0000 7f0f0118f640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 -- 172.21.15.23:0/1325545795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 msgr2=0x5559d21f96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 -- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 msgr2=0x5559d22006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d22006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 -- 172.21.15.23:0/1325545795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec0d0e70 con 0x5559d21eef80 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f02191640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d22006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f01990640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009900 tx=0x7f0ef802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d684647fc1b5f988 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80054d0 con 0x5559d21eef80 2024-03-16T20:35:34.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0ef8005630 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559d1e566c0 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef80035f0 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 10 client.15243 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5559d21552b0 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5559d1e566c0 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0ef803c040 con 0x5559d21eef80 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 10 client.15243 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 10 client.15243 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 20 client.15243 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.038+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2205ad0 0x5559d2204d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.039+0000 7f0f067ce740 4 client.15243 mds_command: new command op to 24416 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.039+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 --> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559d1ea7270 con 0x5559d2205ad0 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.039+0000 7f0f0118f640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2205ad0 0x5559d2204d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.039+0000 7f0ee57fa640 10 client.15243 ms_handle_connect on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.039+0000 7f0f0118f640 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2205ad0 0x5559d2204d60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=348bea64c9642965 server_cookie=fb29403cb5cf9c66 in_seq=0 out_seq=0 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0ee57fa640 1 -- 172.21.15.23:0/1325545795 <== mds.4 v2:172.21.15.31:6832/1919282692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5559d1ea7270 con 0x5559d2205ad0 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0ee57fa640 10 client.15243 handle_command_reply: tid=0 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 client.15243 shutdown 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 msgr2=0x5559d21f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009900 tx=0x7f0ef802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 shutdown_connections 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2205ad0 0x5559d2204d60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21f4ae0 0x5559d22006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21f96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 --2- 172.21.15.23:0/1325545795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21eef80 0x5559d21f9160 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0ee57fa640 0 client.15243 ms_handle_reset on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 >> 172.21.15.23:0/1325545795 conn(0x5559d21ea4f0 msgr2=0x5559d21e97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 shutdown_connections 2024-03-16T20:35:34.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.040+0000 7f0f067ce740 1 -- 172.21.15.23:0/1325545795 wait complete. 2024-03-16T20:35:34.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.041+0000 7f0f067ce740 20 client.15243 trim_cache size 0 max 16384 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 Processor -- start 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 -- start start 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d20dc490 con 0x5559d21f4ae0 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d21f7630 con 0x5559d21eef80 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.057+0000 7f0f067ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d1ea7270 con 0x5559d21ef970 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f02191640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f01990640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:40970/0 (socket says 172.21.15.23:40970) 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f01990640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21f8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39474/0 (socket says 172.21.15.23:39474) 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 -- 172.21.15.23:0/8666859 learned_addr learned my addr 172.21.15.23:0/8666859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 -- 172.21.15.23:0/8666859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 msgr2=0x5559d21f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21f8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 -- 172.21.15.23:0/8666859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 msgr2=0x5559d21f92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21f92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 -- 172.21.15.23:0/8666859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d20f1a20 con 0x5559d21ef970 2024-03-16T20:35:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f02191640 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.058+0000 7f0f0118f640 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0eec096b10 tx=0x7f0eec0d3550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24cf6754c5d4c4fb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0ee77fe640 1 -- 172.21.15.23:0/8666859 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0cdb80 con 0x5559d21ef970 2024-03-16T20:35:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0ee77fe640 1 -- 172.21.15.23:0/8666859 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0eec09a480 con 0x5559d21ef970 2024-03-16T20:35:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0ee77fe640 1 -- 172.21.15.23:0/8666859 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec09a6b0 con 0x5559d21ef970 2024-03-16T20:35:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 -- 172.21.15.23:0/8666859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 msgr2=0x5559d22006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0eec096b10 tx=0x7f0eec0d3550 comp rx=0 tx=0).stop 2024-03-16T20:35:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 -- 172.21.15.23:0/8666859 shutdown_connections 2024-03-16T20:35:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d22006a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 --2- 172.21.15.23:0/8666859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21f8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 -- 172.21.15.23:0/8666859 >> 172.21.15.23:0/8666859 conn(0x5559d21ea4f0 msgr2=0x5559d21ea8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 -- 172.21.15.23:0/8666859 shutdown_connections 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.062+0000 7f0f067ce740 1 -- 172.21.15.23:0/8666859 wait complete. 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 -- start start 2024-03-16T20:35:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 Processor -- start 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21fc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21fce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d2200310 con 0x5559d21f4ae0 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d2200480 con 0x5559d21eef80 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d2200640 con 0x5559d21ef970 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f067ce740 10 client.0 _ll_register_callbacks cb 0x7f0ef8044050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41772/0 (socket says 172.21.15.23:41772) 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 -- 172.21.15.23:0/3137336861 learned_addr learned my addr 172.21.15.23:0/3137336861 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f01990640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21fc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f0118f640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21fce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 -- 172.21.15.23:0/3137336861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 msgr2=0x5559d21fce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21fce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 -- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 msgr2=0x5559d21fc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.063+0000 7f0f02191640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21fc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f02191640 1 -- 172.21.15.23:0/3137336861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef4011790 con 0x5559d21f4ae0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f0118f640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f02191640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4010790 tx=0x7f0ef4010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ccfab9878f15c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4016070 con 0x5559d21f4ae0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0eec09a1f0 con 0x5559d21f4ae0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0ef4003d80 con 0x5559d21f4ae0 2024-03-16T20:35:34.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef400f8c0 con 0x5559d21f4ae0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 10 client.15249 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5559d210bf20 con 0x5559d21f4ae0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0eec09a1f0 con 0x5559d21f4ae0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0ef4004670 con 0x5559d21f4ae0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 10 client.15249 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 10 client.15249 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 20 client.15249 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2265950 0x5559d2261d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 4 client.15249 mds_command: new command op to 24416 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.064+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 --> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5559d1e93610 con 0x5559d2265950 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f01990640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2265950 0x5559d2261d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f01990640 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2265950 0x5559d2261d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dd123bde3beb9d12 server_cookie=3a74feb3442839a6 in_seq=0 out_seq=0 2024-03-16T20:35:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0ee57fa640 10 client.15249 ms_handle_connect on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0ee57fa640 1 -- 172.21.15.23:0/3137336861 <== mds.4 v2:172.21.15.31:6832/1919282692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5559d1e93610 con 0x5559d2265950 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0ee57fa640 10 client.15249 handle_command_reply: tid=0 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 client.15249 shutdown 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 msgr2=0x5559d21ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4010790 tx=0x7f0ef4010c50 comp rx=0 tx=0).stop 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 shutdown_connections 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559d21f4ae0 0x5559d21ffdd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559d21ef970 0x5559d21fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5559d2265950 0x5559d2261d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 --2- 172.21.15.23:0/3137336861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d21eef80 0x5559d21fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0ee57fa640 0 client.15249 ms_handle_reset on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.065+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 >> 172.21.15.23:0/3137336861 conn(0x5559d21ea4f0 msgr2=0x5559d21f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 shutdown_connections 2024-03-16T20:35:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f067ce740 1 -- 172.21.15.23:0/3137336861 wait complete. 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f067ce740 20 client.15249 trim_cache size 0 max 16384 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24416, 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 84.133655093000002, 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 137.73101195699999 2024-03-16T20:35:34.077 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 msgr2=0x7f0ee80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 0x7f0ee80a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005ab0 tx=0x7f0ef8005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 msgr2=0x7f0efc19c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00e9f0 tx=0x7f0eec00edf0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 shutdown_connections 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0ee80a1a30 0x7f0ee80a3ef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0efc101fc0 0x7f0efc19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0efc101600 0x7f0efc19c300 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 --2- 172.21.15.23:0/2974266814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0efc100410 0x7f0efc06a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 >> 172.21.15.23:0/2974266814 conn(0x7f0efc0fb9a0 msgr2=0x7f0efc1049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.066+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 shutdown_connections 2024-03-16T20:35:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.067+0000 7f0f03bf4640 1 -- 172.21.15.23:0/2974266814 wait complete. 2024-03-16T20:35:34.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-16T20:35:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 -- 172.21.15.23:0/3917036402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 msgr2=0x7f86d8106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/3917036402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8106cf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009900 tx=0x7f86c802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 -- 172.21.15.23:0/3917036402 shutdown_connections 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/3917036402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 0x7f86d8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/3917036402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8106cf0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/3917036402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 0x7f86d81043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 -- 172.21.15.23:0/3917036402 >> 172.21.15.23:0/3917036402 conn(0x7f86d80fb980 msgr2=0x7f86d80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 -- 172.21.15.23:0/3917036402 shutdown_connections 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.180+0000 7f86dfb1e640 1 -- 172.21.15.23:0/3917036402 wait complete. 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 Processor -- start 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 -- start start 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 0x7f86d8197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 0x7f86d819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819f080 con 0x7f86d8104900 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819f1f0 con 0x7f86d8101fd0 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dfb1e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819f3b0 con 0x7f86d8107230 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dd0b9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dd0b9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41794/0 (socket says 172.21.15.23:41794) 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dd0b9640 1 -- 172.21.15.23:0/1107285150 learned_addr learned my addr 172.21.15.23:0/1107285150 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 0x7f86d8197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 0x7f86d819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.181+0000 7f86dd0b9640 1 -- 172.21.15.23:0/1107285150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 msgr2=0x7f86d819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 0x7f86d819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dd0b9640 1 -- 172.21.15.23:0/1107285150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 msgr2=0x7f86d8197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 0x7f86d8197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dd0b9640 1 -- 172.21.15.23:0/1107285150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8009560 con 0x7f86d8104900 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f86c8005ab0 tx=0x7f86c8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86ceffd640 1 -- 172.21.15.23:0/1107285150 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80043d0 con 0x7f86d8104900 2024-03-16T20:35:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86ceffd640 1 -- 172.21.15.23:0/1107285150 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86c80052d0 con 0x7f86d8104900 2024-03-16T20:35:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d819f630 con 0x7f86d8104900 2024-03-16T20:35:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86ceffd640 1 -- 172.21.15.23:0/1107285150 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8003d40 con 0x7f86d8104900 2024-03-16T20:35:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.182+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d819fb90 con 0x7f86d8104900 2024-03-16T20:35:34.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86ceffd640 1 -- 172.21.15.23:0/1107285150 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f86c8005440 con 0x7f86d8104900 2024-03-16T20:35:34.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 Processor -- start 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 -- start start 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0c9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a078cd0 con 0x557d5a0c1d80 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a02bf20 con 0x557d5a0c6ef0 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86e27d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a029450 con 0x557d5a0c1390 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86dd8ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86dd8ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41808/0 (socket says 172.21.15.23:41808) 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86dd8ba640 1 -- 172.21.15.23:0/2611299928 learned_addr learned my addr 172.21.15.23:0/2611299928 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.185+0000 7f86de0bb640 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86ceffd640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 0x7f86c40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 -- 172.21.15.23:0/2611299928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 msgr2=0x557d5a0c17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0c17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 0x7f86c40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 -- 172.21.15.23:0/2611299928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 msgr2=0x557d5a0c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 -- 172.21.15.23:0/2611299928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d59fd9f20 con 0x557d5a0c1d80 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86ceffd640 1 -- 172.21.15.23:0/1107285150 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f86c80ef7e0 con 0x7f86d8104900 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009cf0 tx=0x7f86d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474ee0e179e1b56d server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86c3fff640 1 -- 172.21.15.23:0/2611299928 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0009ea0 con 0x557d5a0c1d80 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86c3fff640 1 -- 172.21.15.23:0/2611299928 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86d0004230 con 0x557d5a0c1d80 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 0x7f86c40a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557d5a0c25e0 tx=0x7f86d4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.186+0000 7f86c3fff640 1 -- 172.21.15.23:0/2611299928 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0019880 con 0x557d5a0c1d80 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- 172.21.15.23:0/2611299928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 msgr2=0x557d5a0c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009cf0 tx=0x7f86d0011810 comp rx=0 tx=0).stop 2024-03-16T20:35:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- 172.21.15.23:0/2611299928 shutdown_connections 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0c6880 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2611299928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0c17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- 172.21.15.23:0/2611299928 >> 172.21.15.23:0/2611299928 conn(0x557d5a0b84f0 msgr2=0x557d5a0b88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- 172.21.15.23:0/2611299928 shutdown_connections 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- 172.21.15.23:0/2611299928 wait complete. 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 -- start start 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.187+0000 7f86e27d1740 1 Processor -- start 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a024c30 con 0x557d5a0c1390 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d59fc9d30 con 0x557d5a0c6ef0 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d59cf8310 con 0x557d5a0c1d80 2024-03-16T20:35:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 10 client.0 _ll_register_callbacks cb 0x557d59fd1d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41818/0 (socket says 172.21.15.23:41818) 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 -- 172.21.15.23:0/4103143622 learned_addr learned my addr 172.21.15.23:0/4103143622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86de0bb640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 -- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 msgr2=0x557d5a0cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 -- 172.21.15.23:0/4103143622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 msgr2=0x557d5a0d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0d2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 -- 172.21.15.23:0/4103143622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d0009950 con 0x557d5a0c1390 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86d001b040 tx=0x7f86d00096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1538fb384e4e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0005040 con 0x557d5a0c1390 2024-03-16T20:35:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f86d00051a0 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.188+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d59d246c0 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86d0005360 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557d5a0232b0 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557d59d246c0 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f86d0021040 con 0x557d5a0c1390 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 10 client.15261 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 20 client.15261 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a0d7e30 0x557d5a0d70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 4 client.15261 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 --> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d59d75270 con 0x557d5a0d7e30 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.189+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a0d7e30 0x557d5a0d70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86c17fa640 10 client.15261 ms_handle_connect on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a0d7e30 0x557d5a0d70c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a28d866368bc016 server_cookie=d573ea1a8450a699 in_seq=0 out_seq=0 2024-03-16T20:35:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86c17fa640 1 -- 172.21.15.23:0/4103143622 <== mds.3 v2:172.21.15.189:6832/1083314774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d59d75270 con 0x557d5a0d7e30 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86c17fa640 10 client.15261 handle_command_reply: tid=0 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 client.15261 shutdown 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 msgr2=0x557d5a0cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86d001b040 tx=0x7f86d00096c0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 shutdown_connections 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a0d7e30 0x557d5a0d70c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0d2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 --2- 172.21.15.23:0/4103143622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1390 0x557d5a0cb600 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86c17fa640 0 client.15261 ms_handle_reset on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.190+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 >> 172.21.15.23:0/4103143622 conn(0x557d5a0b84f0 msgr2=0x557d5a0b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.191+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 shutdown_connections 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.191+0000 7f86e27d1740 1 -- 172.21.15.23:0/4103143622 wait complete. 2024-03-16T20:35:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.191+0000 7f86e27d1740 20 client.15261 trim_cache size 0 max 16384 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 Processor -- start 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 -- start start 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0b7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d59faa490 con 0x557d5a0c1d80 2024-03-16T20:35:34.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a0b8f40 con 0x557d5a0c6ef0 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86e27d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d59d75270 con 0x557d5a0c1390 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41834/0 (socket says 172.21.15.23:41834) 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 -- 172.21.15.23:0/2533441859 learned_addr learned my addr 172.21.15.23:0/2533441859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86de0bb640 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 -- 172.21.15.23:0/2533441859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 msgr2=0x557d5a0b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 -- 172.21.15.23:0/2533441859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 msgr2=0x557d5a0b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 -- 172.21.15.23:0/2533441859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d59fbfa20 con 0x557d5a0c1d80 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004650 tx=0x7f86c80dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295a4084231adfdf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.208+0000 7f86c37fe640 1 -- 172.21.15.23:0/2533441859 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80bc1e0 con 0x557d5a0c1d80 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86c37fe640 1 -- 172.21.15.23:0/2533441859 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86c80bbf50 con 0x557d5a0c1d80 2024-03-16T20:35:34.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86c37fe640 1 -- 172.21.15.23:0/2533441859 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c80c4bb0 con 0x557d5a0c1d80 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- 172.21.15.23:0/2533441859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 msgr2=0x557d5a0d2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004650 tx=0x7f86c80dc040 comp rx=0 tx=0).stop 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- 172.21.15.23:0/2533441859 shutdown_connections 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c6ef0 0x557d5a0b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2ad0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 --2- 172.21.15.23:0/2533441859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1390 0x557d5a0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- 172.21.15.23:0/2533441859 >> 172.21.15.23:0/2533441859 conn(0x557d5a0b84f0 msgr2=0x557d5a0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- 172.21.15.23:0/2533441859 shutdown_connections 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- 172.21.15.23:0/2533441859 wait complete. 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.209+0000 7f86e27d1740 1 -- start start 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 Processor -- start 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 0x557d5a0d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a0cf4a0 con 0x557d5a0c6ef0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a0cf610 con 0x557d5a0c1390 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5a0cf7d0 con 0x557d5a0c1d80 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86e27d1740 10 client.0 _ll_register_callbacks cb 0x7f86c80b84e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86dd8ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 0x557d5a0d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41848/0 (socket says 172.21.15.23:41848) 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 -- 172.21.15.23:0/1237005818 learned_addr learned my addr 172.21.15.23:0/1237005818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 -- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 msgr2=0x557d5a0d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 -- 172.21.15.23:0/1237005818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 msgr2=0x557d5a0d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 0x557d5a0d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86de0bb640 1 -- 172.21.15.23:0/1237005818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d819fb90 con 0x557d5a0c6ef0 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 0x557d5a0d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.210+0000 7f86dd0b9640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86de0bb640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f86d4003ec0 tx=0x7f86d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df2cb888659b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4027070 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f86d4004550 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d5a0d7b40 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86d4004710 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 10 client.15267 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557d59fe5780 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557d5a0d7b40 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f86d4005740 con 0x557d5a0c6ef0 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 10 client.15267 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 10 client.15267 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 20 client.15267 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a137e60 0x557d5a1342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 4 client.15267 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 --> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f86d0004230 con 0x557d5a137e60 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.211+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a137e60 0x557d5a1342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86dd8ba640 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a137e60 0x557d5a1342c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83dc8652f67d9fcb server_cookie=20f96543a6e97522 in_seq=0 out_seq=0 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86c17fa640 10 client.15267 ms_handle_connect on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86c17fa640 1 -- 172.21.15.23:0/1237005818 <== mds.3 v2:172.21.15.189:6832/1083314774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f86d0004230 con 0x557d5a137e60 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86c17fa640 10 client.15267 handle_command_reply: tid=0 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 client.15267 shutdown 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 msgr2=0x557d5a0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f86d4003ec0 tx=0x7f86d4004a40 comp rx=0 tx=0).stop 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 shutdown_connections 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x557d5a137e60 0x557d5a1342c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5a0c6ef0 0x557d5a0cee60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557d5a0c1d80 0x557d5a0d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 --2- 172.21.15.23:0/1237005818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d5a0c1390 0x557d5a0d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86c17fa640 0 client.15267 ms_handle_reset on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 >> 172.21.15.23:0/1237005818 conn(0x557d5a0b84f0 msgr2=0x557d5a0bb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 shutdown_connections 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 1 -- 172.21.15.23:0/1237005818 wait complete. 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.212+0000 7f86e27d1740 20 client.15267 trim_cache size 0 max 16384 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24418, 2024-03-16T20:35:34.224 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 85.288316909000002, 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 140.67923699100001 2024-03-16T20:35:34.225 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 msgr2=0x7f86c40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 0x7f86c40a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557d5a0c25e0 tx=0x7f86d4004ab0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 msgr2=0x7f86d8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f86c8005ab0 tx=0x7f86c8005e50 comp rx=0 tx=0).stop 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 shutdown_connections 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f86c40a1ab0 0x7f86c40a3f70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86d8107230 0x7f86d819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86d8104900 0x7f86d8197ad0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 --2- 172.21.15.23:0/1107285150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d8101fd0 0x7f86d8197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 >> 172.21.15.23:0/1107285150 conn(0x7f86d80fb980 msgr2=0x7f86d8102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 shutdown_connections 2024-03-16T20:35:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.213+0000 7f86dfb1e640 1 -- 172.21.15.23:0/1107285150 wait complete. 2024-03-16T20:35:34.227 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:34 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3313823357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.227 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:34 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1543123002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.227 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:34 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2272437868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.227 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:34 smithi023 ceph-mon[31879]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-16T20:35:34.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T20:35:34.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 -- 172.21.15.23:0/4024607277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010104900 msgr2=0x7f3010106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 --2- 172.21.15.23:0/4024607277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010104900 0x7f3010106cf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009900 tx=0x7f2ff802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 -- 172.21.15.23:0/4024607277 shutdown_connections 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 --2- 172.21.15.23:0/4024607277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f3010109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 --2- 172.21.15.23:0/4024607277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010104900 0x7f3010106cf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 --2- 172.21.15.23:0/4024607277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010101fd0 0x7f30101043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.327+0000 7f30159a0640 1 -- 172.21.15.23:0/4024607277 >> 172.21.15.23:0/4024607277 conn(0x7f30100fb980 msgr2=0x7f30100fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- 172.21.15.23:0/4024607277 shutdown_connections 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- 172.21.15.23:0/4024607277 wait complete. 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 Processor -- start 2024-03-16T20:35:34.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- start start 2024-03-16T20:35:34.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 0x7f3010197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f301019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019f0e0 con 0x7f3010107230 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019f250 con 0x7f3010101fd0 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.328+0000 7f30159a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019f410 con 0x7f3010104900 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f301019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39532/0 (socket says 172.21.15.23:39532) 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 -- 172.21.15.23:0/2387845260 learned_addr learned my addr 172.21.15.23:0/2387845260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 0x7f3010197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 -- 172.21.15.23:0/2387845260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 msgr2=0x7f3010197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 0x7f3010197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 -- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 msgr2=0x7f301019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f301019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 -- 172.21.15.23:0/2387845260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009840 con 0x7f3010101fd0 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300f7fe640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f301019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300e7fc640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 0x7f3010197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f300effd640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f300800e9f0 tx=0x7f300800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f301499e640 1 -- 172.21.15.23:0/2387845260 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008009700 con 0x7f3010101fd0 2024-03-16T20:35:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff8009560 con 0x7f3010101fd0 2024-03-16T20:35:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f301499e640 1 -- 172.21.15.23:0/2387845260 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f300800ccc0 con 0x7f3010101fd0 2024-03-16T20:35:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f301499e640 1 -- 172.21.15.23:0/2387845260 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008018820 con 0x7f3010101fd0 2024-03-16T20:35:34.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.329+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301019fa90 con 0x7f3010101fd0 2024-03-16T20:35:34.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.331+0000 7f301499e640 1 -- 172.21.15.23:0/2387845260 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3008010430 con 0x7f3010101fd0 2024-03-16T20:35:34.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 Processor -- start 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 -- start start 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f50970 0x5591c3f55470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3f0bcd0 con 0x5591c3f55ae0 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3ebef20 con 0x5591c3f50970 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f3018576740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3ebc450 con 0x5591c3f4ff80 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41886/0 (socket says 172.21.15.23:41886) 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 -- 172.21.15.23:0/378316734 learned_addr learned my addr 172.21.15.23:0/378316734 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300f7fe640 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300effd640 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f50970 0x5591c3f55470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f301499e640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 0x7f2ffc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300f7fe640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 0x7f2ffc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 -- 172.21.15.23:0/378316734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 msgr2=0x5591c3f503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 -- 172.21.15.23:0/378316734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f50970 msgr2=0x5591c3f55470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f50970 0x5591c3f55470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 -- 172.21.15.23:0/378316734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591c3e6cf20 con 0x5591c3f55ae0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f301499e640 1 -- 172.21.15.23:0/2387845260 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3008014070 con 0x7f3010101fd0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.332+0000 7f300e7fc640 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3010069150 tx=0x7f2ff802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb71d2098d64f46b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f300f7fe640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 0x7f2ffc0a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5591c3f511d0 tx=0x7f3000004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3005ffb640 1 -- 172.21.15.23:0/378316734 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8004780 con 0x5591c3f55ae0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3005ffb640 1 -- 172.21.15.23:0/378316734 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff8004510 con 0x5591c3f55ae0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3005ffb640 1 -- 172.21.15.23:0/378316734 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8005500 con 0x5591c3f55ae0 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 -- 172.21.15.23:0/378316734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 msgr2=0x5591c3f57f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3010069150 tx=0x7f2ff802cd20 comp rx=0 tx=0).stop 2024-03-16T20:35:34.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 -- 172.21.15.23:0/378316734 shutdown_connections 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f55ae0 0x5591c3f57f70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f50970 0x5591c3f55470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 --2- 172.21.15.23:0/378316734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 -- 172.21.15.23:0/378316734 >> 172.21.15.23:0/378316734 conn(0x5591c3f4b4f0 msgr2=0x5591c3f4b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 -- 172.21.15.23:0/378316734 shutdown_connections 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.333+0000 7f3018576740 1 -- 172.21.15.23:0/378316734 wait complete. 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 -- start start 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 Processor -- start 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f50970 0x5591c3f5a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f61690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3eb7c30 con 0x5591c3f4ff80 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3e5cd30 con 0x5591c3f55ae0 2024-03-16T20:35:34.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3b8b310 con 0x5591c3f50970 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f3018576740 10 client.0 _ll_register_callbacks cb 0x5591c3e64d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41890/0 (socket says 172.21.15.23:41890) 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 -- 172.21.15.23:0/326870103 learned_addr learned my addr 172.21.15.23:0/326870103 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300e7fc640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f50970 0x5591c3f5a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300f7fe640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f61690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 -- 172.21.15.23:0/326870103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f50970 msgr2=0x5591c3f5a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f50970 0x5591c3f5a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 -- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 msgr2=0x5591c3f61690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f61690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.334+0000 7f300effd640 1 -- 172.21.15.23:0/326870103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301010a200 con 0x5591c3f4ff80 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f300effd640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f30080cb6a0 tx=0x7f30080d1060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18d69a40ff2284f server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30080ba2e0 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3008010a10 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591c3bb76c0 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f300809a100 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5591c3eb62b0 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5591c3bb76c0 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3008099bf0 con 0x5591c3f4ff80 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 10 client.15279 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 20 client.15279 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3f66a60 0x5591c3f6aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 4 client.15279 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.335+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 --> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591c3c08270 con 0x5591c3f66a60 2024-03-16T20:35:34.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.336+0000 7f300e7fc640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3f66a60 0x5591c3f6aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.336+0000 7f300e7fc640 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3f66a60 0x5591c3f6aee0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ccaf3f525812e7cb server_cookie=ac94211cafc5e5f3 in_seq=0 out_seq=0 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.336+0000 7f2feeffd640 10 client.15279 ms_handle_connect on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f2feeffd640 1 -- 172.21.15.23:0/326870103 <== mds.2 v2:172.21.15.31:6836/424329834 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5591c3c08270 con 0x5591c3f66a60 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f2feeffd640 10 client.15279 handle_command_reply: tid=0 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 client.15279 shutdown 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 msgr2=0x5591c3f5a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f30080cb6a0 tx=0x7f30080d1060 comp rx=0 tx=0).stop 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 shutdown_connections 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3f66a60 0x5591c3f6aee0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f61690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f50970 0x5591c3f5a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f2feeffd640 0 client.15279 ms_handle_reset on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 --2- 172.21.15.23:0/326870103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f4ff80 0x5591c3f5a110 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 >> 172.21.15.23:0/326870103 conn(0x5591c3f4b4f0 msgr2=0x5591c3f4a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 shutdown_connections 2024-03-16T20:35:34.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 1 -- 172.21.15.23:0/326870103 wait complete. 2024-03-16T20:35:34.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.337+0000 7f3018576740 20 client.15279 trim_cache size 0 max 16384 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 Processor -- start 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 -- start start 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f4b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f5a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3e3d490 con 0x5591c3f50970 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3f218c0 con 0x5591c3f55ae0 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f3018576740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3e70f40 con 0x5591c3f4ff80 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41902/0 (socket says 172.21.15.23:41902) 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 -- 172.21.15.23:0/4160074137 learned_addr learned my addr 172.21.15.23:0/4160074137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300effd640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f4b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300f7fe640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f5a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 -- 172.21.15.23:0/4160074137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 msgr2=0x5591c3f4b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f4b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 -- 172.21.15.23:0/4160074137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 msgr2=0x5591c3f5a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f5a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 -- 172.21.15.23:0/4160074137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591c3e52a20 con 0x5591c3f50970 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300f7fe640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f5a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:35:34.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f300e7fc640 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x5591c3f4c460 tx=0x7f2ff8042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb84b47f9cbfd00 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f30057fa640 1 -- 172.21.15.23:0/4160074137 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80046c0 con 0x5591c3f50970 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f30057fa640 1 -- 172.21.15.23:0/4160074137 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff8005450 con 0x5591c3f50970 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.354+0000 7f30057fa640 1 -- 172.21.15.23:0/4160074137 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8052d40 con 0x5591c3f50970 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- 172.21.15.23:0/4160074137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 msgr2=0x5591c3f59d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x5591c3f4c460 tx=0x7f2ff8042040 comp rx=0 tx=0).stop 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- 172.21.15.23:0/4160074137 shutdown_connections 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3f5a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3f59d20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 --2- 172.21.15.23:0/4160074137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3f4b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- 172.21.15.23:0/4160074137 >> 172.21.15.23:0/4160074137 conn(0x5591c3f4b4f0 msgr2=0x5591c3f51390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- 172.21.15.23:0/4160074137 shutdown_connections 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- 172.21.15.23:0/4160074137 wait complete. 2024-03-16T20:35:34.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 -- start start 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.355+0000 7f3018576740 1 Processor -- start 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3fbfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3fc4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3fc5210 con 0x5591c3f50970 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3fc5380 con 0x5591c3f55ae0 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c3fc5540 con 0x5591c3f4ff80 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 10 client.0 _ll_register_callbacks cb 0x7f3008010c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41912/0 (socket says 172.21.15.23:41912) 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 -- 172.21.15.23:0/3432252724 learned_addr learned my addr 172.21.15.23:0/3432252724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300effd640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3fbfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300f7fe640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3fc4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 -- 172.21.15.23:0/3432252724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 msgr2=0x5591c3fbfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3fbfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 -- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 msgr2=0x5591c3fc4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3fc4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 -- 172.21.15.23:0/3432252724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591c3e6cf20 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f300e7fc640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2ff802c170 tx=0x7f2ff8052880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bb90f391663cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f2feeffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80046c0 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f2ff8052590 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.356+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591c3f67a90 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ff8004f70 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 10 client.15285 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5591c3e78780 con 0x5591c3f50970 2024-03-16T20:35:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5591c3f67a90 con 0x5591c3f50970 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f2ff803c050 con 0x5591c3f50970 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 10 client.15285 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 20 client.15285 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3fca390 0x5591c3fc6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 4 client.15285 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 --> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f2ff8004510 con 0x5591c3fca390 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f300effd640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3fca390 0x5591c3fc6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f300effd640 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3fca390 0x5591c3fc6830 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a69e700dbd260480 server_cookie=707428357228bb8f in_seq=0 out_seq=0 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.357+0000 7f2feeffd640 10 client.15285 ms_handle_connect on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f2feeffd640 1 -- 172.21.15.23:0/3432252724 <== mds.2 v2:172.21.15.31:6836/424329834 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f2ff8004510 con 0x5591c3fca390 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f2feeffd640 10 client.15285 handle_command_reply: tid=0 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 client.15285 shutdown 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 msgr2=0x5591c3fc2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2ff802c170 tx=0x7f2ff8052880 comp rx=0 tx=0).stop 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 shutdown_connections 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5591c3fca390 0x5591c3fc6830 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591c3f55ae0 0x5591c3fc4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5591c3f50970 0x5591c3fc2540 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f2feeffd640 0 client.15285 ms_handle_reset on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 --2- 172.21.15.23:0/3432252724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591c3f4ff80 0x5591c3fbfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 >> 172.21.15.23:0/3432252724 conn(0x5591c3f4b4f0 msgr2=0x5591c3f4b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 shutdown_connections 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 1 -- 172.21.15.23:0/3432252724 wait complete. 2024-03-16T20:35:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f3018576740 20 client.15285 trim_cache size 0 max 16384 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24493, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 85.640997940999995, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 121.179187111 2024-03-16T20:35:34.371 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:34.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 msgr2=0x7f2ffc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 0x7f2ffc0a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5591c3f511d0 tx=0x7f3000004ab0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 msgr2=0x7f30101975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f300800e9f0 tx=0x7f300800edf0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 shutdown_connections 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2ffc0a1a30 0x7f2ffc0a3ef0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010107230 0x7f301019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010104900 0x7f3010197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 --2- 172.21.15.23:0/2387845260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3010101fd0 0x7f30101975f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.358+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 >> 172.21.15.23:0/2387845260 conn(0x7f30100fb980 msgr2=0x7f3010102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.359+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 shutdown_connections 2024-03-16T20:35:34.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.359+0000 7f30159a0640 1 -- 172.21.15.23:0/2387845260 wait complete. 2024-03-16T20:35:34.374 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T20:35:34.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:34 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3313823357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:34 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1543123002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:34 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2272437868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:34 smithi031 ceph-mon[36281]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-16T20:35:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:34 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3313823357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:34 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1543123002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:34 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2272437868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:34.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:34 smithi189 ceph-mon[36655]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- 172.21.15.23:0/1917081028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458101fc0 msgr2=0x7f4458108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- 172.21.15.23:0/1917081028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458101fc0 0x7f4458108850 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4450009ca0 tx=0x7f445002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- 172.21.15.23:0/1917081028 shutdown_connections 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- 172.21.15.23:0/1917081028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458101fc0 0x7f4458108850 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- 172.21.15.23:0/1917081028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 0x7f4458101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- 172.21.15.23:0/1917081028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458100410 0x7f4458100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- 172.21.15.23:0/1917081028 >> 172.21.15.23:0/1917081028 conn(0x7f44580fb9a0 msgr2=0x7f44580fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- 172.21.15.23:0/1917081028 shutdown_connections 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- 172.21.15.23:0/1917081028 wait complete. 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 Processor -- start 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- start start 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 0x7f4458197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458101fc0 0x7f445819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445819f090 con 0x7f4458100410 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445819f200 con 0x7f4458101fc0 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.473+0000 7f445fc23640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445819f3c0 con 0x7f4458101600 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41940/0 (socket says 172.21.15.23:41940) 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 -- 172.21.15.23:0/2844803525 learned_addr learned my addr 172.21.15.23:0/2844803525 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d1be640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 0x7f4458197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458101fc0 0x7f445819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 -- 172.21.15.23:0/2844803525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 msgr2=0x7f4458197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 0x7f4458197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 -- 172.21.15.23:0/2844803525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458101fc0 msgr2=0x7f445819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458101fc0 0x7f445819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 -- 172.21.15.23:0/2844803525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454009490 con 0x7f4458100410 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f445400e9c0 tx=0x7f445400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f444effd640 1 -- 172.21.15.23:0/2844803525 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445400cde0 con 0x7f4458100410 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f444effd640 1 -- 172.21.15.23:0/2844803525 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4454010910 con 0x7f4458100410 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4450009900 con 0x7f4458100410 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f444effd640 1 -- 172.21.15.23:0/2844803525 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454018a10 con 0x7f4458100410 2024-03-16T20:35:34.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.474+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445819fa20 con 0x7f4458100410 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f444effd640 1 -- 172.21.15.23:0/2844803525 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4454010a80 con 0x7f4458100410 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 Processor -- start 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 -- start start 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d604470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d606f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d5bacd0 con 0x56413d5fef80 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d56df20 con 0x56413d5ff970 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.477+0000 7f4462800740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d56b450 con 0x56413d604ae0 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41946/0 (socket says 172.21.15.23:41946) 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 -- 172.21.15.23:0/2428701312 learned_addr learned my addr 172.21.15.23:0/2428701312 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445d1be640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d606f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d604470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f444effd640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 0x7f44440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 -- 172.21.15.23:0/2428701312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 msgr2=0x56413d606f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d606f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 -- 172.21.15.23:0/2428701312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 msgr2=0x56413d604470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d604470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 -- 172.21.15.23:0/2428701312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56413d51bf20 con 0x56413d5fef80 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445d1be640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d606f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 0x7f44440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f444effd640 1 -- 172.21.15.23:0/2844803525 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4454014070 con 0x7f4458100410 2024-03-16T20:35:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f445002c700 tx=0x7f4450005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43800806071eafa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f4443fff640 1 -- 172.21.15.23:0/2428701312 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44500347c0 con 0x56413d5fef80 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f4443fff640 1 -- 172.21.15.23:0/2428701312 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44500329b0 con 0x56413d5fef80 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.478+0000 7f4443fff640 1 -- 172.21.15.23:0/2428701312 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450004240 con 0x56413d5fef80 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 -- 172.21.15.23:0/2428701312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 msgr2=0x56413d5ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 0x7f44440a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f445004d000 tx=0x7f445004a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f445002c700 tx=0x7f4450005f40 comp rx=0 tx=0).stop 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 -- 172.21.15.23:0/2428701312 shutdown_connections 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d606f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d604470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 --2- 172.21.15.23:0/2428701312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d5ff3a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 -- 172.21.15.23:0/2428701312 >> 172.21.15.23:0/2428701312 conn(0x56413d5fa4f0 msgr2=0x56413d5fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 -- 172.21.15.23:0/2428701312 shutdown_connections 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.479+0000 7f4462800740 1 -- 172.21.15.23:0/2428701312 wait complete. 2024-03-16T20:35:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 -- start start 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 Processor -- start 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5ff970 0x56413d6096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d604ae0 0x56413d610710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d566c30 con 0x56413d5fef80 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d50bd30 con 0x56413d604ae0 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d23a310 con 0x56413d5ff970 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f4462800740 10 client.0 _ll_register_callbacks cb 0x56413d513d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41960/0 (socket says 172.21.15.23:41960) 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 -- 172.21.15.23:0/2223032309 learned_addr learned my addr 172.21.15.23:0/2223032309 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d1be640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5ff970 0x56413d6096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445e1c0640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d604ae0 0x56413d610710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 -- 172.21.15.23:0/2223032309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5ff970 msgr2=0x56413d6096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5ff970 0x56413d6096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 -- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d604ae0 msgr2=0x56413d610710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d604ae0 0x56413d610710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.480+0000 7f445d9bf640 1 -- 172.21.15.23:0/2223032309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4458109150 con 0x56413d5fef80 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f445d9bf640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4454096ea0 tx=0x7f4454095410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedd0ad5a87322c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445409a580 con 0x56413d5fef80 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f445409b140 con 0x56413d5fef80 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56413d2666c0 con 0x56413d5fef80 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f445409a3a0 con 0x56413d5fef80 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56413d5652b0 con 0x56413d5fef80 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44540b2070 con 0x56413d5fef80 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f44540d4060 con 0x56413d5fef80 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 10 client.15297 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 20 client.15297 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d615b70 0x56413d614e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 4 client.15297 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 --> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56413d2b7270 con 0x56413d615b70 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.481+0000 7f445d1be640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d615b70 0x56413d614e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.482+0000 7f445d1be640 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d615b70 0x56413d614e30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc89f83919e9fbb5 server_cookie=4777479bb220955a in_seq=0 out_seq=0 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.482+0000 7f44417fa640 10 client.15297 ms_handle_connect on v2:172.21.15.31:6838/37663771 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.482+0000 7f44417fa640 1 -- 172.21.15.23:0/2223032309 <== mds.1 v2:172.21.15.31:6838/37663771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56413d2b7270 con 0x56413d615b70 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.482+0000 7f44417fa640 10 client.15297 handle_command_reply: tid=0 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.482+0000 7f4462800740 1 client.15297 shutdown 2024-03-16T20:35:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 msgr2=0x56413d609190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4454096ea0 tx=0x7f4454095410 comp rx=0 tx=0).stop 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 shutdown_connections 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d615b70 0x56413d614e30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d604ae0 0x56413d610710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5ff970 0x56413d6096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f44417fa640 0 client.15297 ms_handle_reset on v2:172.21.15.31:6838/37663771 2024-03-16T20:35:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 --2- 172.21.15.23:0/2223032309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d609190 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 >> 172.21.15.23:0/2223032309 conn(0x56413d5fa4f0 msgr2=0x56413d5f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 shutdown_connections 2024-03-16T20:35:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 1 -- 172.21.15.23:0/2223032309 wait complete. 2024-03-16T20:35:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.483+0000 7f4462800740 20 client.15297 trim_cache size 0 max 16384 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 Processor -- start 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 -- start start 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d6106a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d6092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d4ec490 con 0x56413d5fef80 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d600060 con 0x56413d5ff970 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f4462800740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d51ff40 con 0x56413d604ae0 2024-03-16T20:35:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d1be640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d6106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41966/0 (socket says 172.21.15.23:41966) 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 -- 172.21.15.23:0/3919367472 learned_addr learned my addr 172.21.15.23:0/3919367472 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445e1c0640 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d6092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 -- 172.21.15.23:0/3919367472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 msgr2=0x56413d6092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d6092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 -- 172.21.15.23:0/3919367472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 msgr2=0x56413d6106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d6106a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 -- 172.21.15.23:0/3919367472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56413d501a20 con 0x56413d5fef80 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f445d9bf640 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f445409a9d0 tx=0x7f44540184b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdef3a915223392b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.500+0000 7f44437fe640 1 -- 172.21.15.23:0/3919367472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44540d0070 con 0x56413d5fef80 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f44437fe640 1 -- 172.21.15.23:0/3919367472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44540b2070 con 0x56413d5fef80 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f44437fe640 1 -- 172.21.15.23:0/3919367472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445409eb90 con 0x56413d5fef80 2024-03-16T20:35:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- 172.21.15.23:0/3919367472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 msgr2=0x56413d608da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f445409a9d0 tx=0x7f44540184b0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- 172.21.15.23:0/3919367472 shutdown_connections 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d604ae0 0x56413d6092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d6106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 --2- 172.21.15.23:0/3919367472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d5fef80 0x56413d608da0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- 172.21.15.23:0/3919367472 >> 172.21.15.23:0/3919367472 conn(0x56413d5fa4f0 msgr2=0x56413d600e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- 172.21.15.23:0/3919367472 shutdown_connections 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- 172.21.15.23:0/3919367472 wait complete. 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 -- start start 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.501+0000 7f4462800740 1 Processor -- start 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5fef80 0x56413d610130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d610670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d60d3f0 con 0x56413d604ae0 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d60d560 con 0x56413d5ff970 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56413d60d720 con 0x56413d5fef80 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f4462800740 10 client.0 _ll_register_callbacks cb 0x7f445409afb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445d1be640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d610670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41982/0 (socket says 172.21.15.23:41982) 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 -- 172.21.15.23:0/4086271319 learned_addr learned my addr 172.21.15.23:0/4086271319 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445d9bf640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5fef80 0x56413d610130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 -- 172.21.15.23:0/4086271319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5fef80 msgr2=0x56413d610130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5fef80 0x56413d610130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 -- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 msgr2=0x56413d610670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d610670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 -- 172.21.15.23:0/4086271319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56413d5652b0 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445d1be640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d610670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f445e1c0640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4450005f10 tx=0x7f4450034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82541232b1d7d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.502+0000 7f44417fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44500347c0 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56413d616ba0 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4450003b80 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4450032b20 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56413d527780 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56413d616ba0 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f445004d440 con 0x56413d604ae0 2024-03-16T20:35:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 10 client.15303 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 20 client.15303 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d675a20 0x56413d671e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 4 client.15303 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 --> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f44500329b0 con 0x56413d675a20 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.503+0000 7f445d9bf640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d675a20 0x56413d671e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f44417fa640 10 client.15303 ms_handle_connect on v2:172.21.15.31:6838/37663771 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f445d9bf640 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d675a20 0x56413d671e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75ee4a0506c3b5a8 server_cookie=2fd0094046df627d in_seq=0 out_seq=0 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f44417fa640 1 -- 172.21.15.23:0/4086271319 <== mds.1 v2:172.21.15.31:6838/37663771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f44500329b0 con 0x56413d675a20 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f44417fa640 10 client.15303 handle_command_reply: tid=0 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 client.15303 shutdown 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 msgr2=0x56413d60ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4450005f10 tx=0x7f4450034d40 comp rx=0 tx=0).stop 2024-03-16T20:35:34.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 shutdown_connections 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x56413d675a20 0x56413d671e80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56413d604ae0 0x56413d60ccf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56413d5ff970 0x56413d610670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 --2- 172.21.15.23:0/4086271319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56413d5fef80 0x56413d610130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f44417fa640 0 client.15303 ms_handle_reset on v2:172.21.15.31:6838/37663771 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 >> 172.21.15.23:0/4086271319 conn(0x56413d5fa4f0 msgr2=0x56413d608d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 shutdown_connections 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 1 -- 172.21.15.23:0/4086271319 wait complete. 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.504+0000 7f4462800740 20 client.15303 trim_cache size 0 max 16384 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:34.516 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24497, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 86.802864357999994, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 109.85527302200001 2024-03-16T20:35:34.517 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:34.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 msgr2=0x7f44440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 0x7f44440a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f445004d000 tx=0x7f445004a000 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 msgr2=0x7f44581975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f445400e9c0 tx=0x7f445400ee80 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 shutdown_connections 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44440a1a30 0x7f44440a3ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4458101fc0 0x7f445819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4458101600 0x7f4458197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 --2- 172.21.15.23:0/2844803525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4458100410 0x7f44581975a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 >> 172.21.15.23:0/2844803525 conn(0x7f44580fb9a0 msgr2=0x7f44580fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 shutdown_connections 2024-03-16T20:35:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.505+0000 7f445fc23640 1 -- 172.21.15.23:0/2844803525 wait complete. 2024-03-16T20:35:34.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 -- 172.21.15.23:0/2977151491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101fc0 msgr2=0x7f0608108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 --2- 172.21.15.23:0/2977151491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101fc0 0x7f0608108850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0600009be0 tx=0x7f060002bef0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 -- 172.21.15.23:0/2977151491 shutdown_connections 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 --2- 172.21.15.23:0/2977151491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101fc0 0x7f0608108850 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 --2- 172.21.15.23:0/2977151491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101600 0x7f0608101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 --2- 172.21.15.23:0/2977151491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 -- 172.21.15.23:0/2977151491 >> 172.21.15.23:0/2977151491 conn(0x7f06080fb9a0 msgr2=0x7f06080fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 -- 172.21.15.23:0/2977151491 shutdown_connections 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.619+0000 7f060f0b5640 1 -- 172.21.15.23:0/2977151491 wait complete. 2024-03-16T20:35:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 Processor -- start 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 -- start start 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 0x7f0608196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101fc0 0x7f060819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060819df30 con 0x7f0608101600 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060819e0a0 con 0x7f0608100410 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060f0b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060819e260 con 0x7f0608101fc0 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f0607fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 0x7f0608196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39644/0 (socket says 172.21.15.23:39644) 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 -- 172.21.15.23:0/673585304 learned_addr learned my addr 172.21.15.23:0/673585304 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060d652640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101fc0 0x7f060819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 -- 172.21.15.23:0/673585304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101fc0 msgr2=0x7f060819d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101fc0 0x7f060819d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 -- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 msgr2=0x7f0608196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 0x7f0608196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 -- 172.21.15.23:0/673585304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0600009840 con 0x7f0608100410 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f0607fff640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 0x7f0608196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f060ce51640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9f0 tx=0x7f05fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.620+0000 7f0605ffb640 1 -- 172.21.15.23:0/673585304 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00ccc0 con 0x7f0608100410 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.621+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060819e540 con 0x7f0608100410 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.621+0000 7f0605ffb640 1 -- 172.21.15.23:0/673585304 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05fc00ce20 con 0x7f0608100410 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.621+0000 7f0605ffb640 1 -- 172.21.15.23:0/673585304 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc010430 con 0x7f0608100410 2024-03-16T20:35:34.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.621+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060819eaa0 con 0x7f0608100410 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0605ffb640 1 -- 172.21.15.23:0/673585304 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f05fc003fb0 con 0x7f0608100410 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 Processor -- start 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 -- start start 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0605ffb640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 0x7f05f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0605ffb640 1 -- 172.21.15.23:0/673585304 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f05fc014070 con 0x7f0608100410 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f0970 0x55911a2f5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f060d652640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 0x7f05f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a2f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a2abcd0 con 0x55911a2eff80 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a25ef20 con 0x55911a2f0970 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.623+0000 7f0611d73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a25c450 con 0x55911a2f5ae0 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42006/0 (socket says 172.21.15.23:42006) 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 -- 172.21.15.23:0/200513883 learned_addr learned my addr 172.21.15.23:0/200513883 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f0607fff640 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f0970 0x55911a2f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060d652640 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a2f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 -- 172.21.15.23:0/200513883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 msgr2=0x55911a2f7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a2f7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 -- 172.21.15.23:0/200513883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f0970 msgr2=0x55911a2f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f0970 0x55911a2f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 -- 172.21.15.23:0/200513883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55911a20cf20 con 0x55911a2eff80 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060d652640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 0x7f05f80a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0600004af0 tx=0x7f060002cad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f060ce51640 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f05fc096fb0 tx=0x7f05fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd86116703cf005 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f05f2ffd640 1 -- 172.21.15.23:0/200513883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc099f40 con 0x55911a2eff80 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f05f2ffd640 1 -- 172.21.15.23:0/200513883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05fc09ace0 con 0x55911a2eff80 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.624+0000 7f05f2ffd640 1 -- 172.21.15.23:0/200513883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0ba0d0 con 0x55911a2eff80 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- 172.21.15.23:0/200513883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 msgr2=0x55911a2f03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f05fc096fb0 tx=0x7f05fc00cf90 comp rx=0 tx=0).stop 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- 172.21.15.23:0/200513883 shutdown_connections 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a2f7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f0970 0x55911a2f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 --2- 172.21.15.23:0/200513883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2eff80 0x55911a2f03a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- 172.21.15.23:0/200513883 >> 172.21.15.23:0/200513883 conn(0x55911a2eb4f0 msgr2=0x55911a2eb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- 172.21.15.23:0/200513883 shutdown_connections 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- 172.21.15.23:0/200513883 wait complete. 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 -- start start 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.625+0000 7f0611d73740 1 Processor -- start 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2fa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a301740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a257c30 con 0x55911a2f0970 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a1fcd30 con 0x55911a2eff80 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559119f2b310 con 0x55911a2f5ae0 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 10 client.0 _ll_register_callbacks cb 0x55911a204d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f060ce51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2fa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f060ce51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2fa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:39660/0 (socket says 172.21.15.23:39660) 2024-03-16T20:35:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f060d652640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a301740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f060ce51640 1 -- 172.21.15.23:0/3034795715 learned_addr learned my addr 172.21.15.23:0/3034795715 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 -- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 msgr2=0x55911a301740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a301740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 -- 172.21.15.23:0/3034795715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 msgr2=0x55911a2fa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2fa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 -- 172.21.15.23:0/3034795715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060819eaa0 con 0x55911a2f0970 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0607fff640 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fa700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55911a2f0d70 tx=0x7f05f400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1303d14bf334b920 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f05e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4017070 con 0x55911a2f0970 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559119f576c0 con 0x55911a2f0970 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.626+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f05f4012110 con 0x55911a2f0970 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f40122d0 con 0x55911a2f0970 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 10 client.15315 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55911a2562b0 con 0x55911a2f0970 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559119f576c0 con 0x55911a2f0970 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f05f4003d10 con 0x55911a2f0970 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 10 client.15315 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 10 client.15315 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 20 client.15315 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a306bf0 0x55911a305e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 4 client.15315 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559119fa8270 con 0x55911a306bf0 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f060ce51640 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a306bf0 0x55911a305e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f05e7fff640 10 client.15315 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.627+0000 7f060ce51640 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a306bf0 0x55911a305e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91f9a921f3b9ab1b server_cookie=7856568c7740adcb in_seq=0 out_seq=0 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f05e7fff640 1 -- 172.21.15.23:0/3034795715 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559119fa8270 con 0x55911a306bf0 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f05e7fff640 10 client.15315 handle_command_reply: tid=0 2024-03-16T20:35:34.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 client.15315 shutdown 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 msgr2=0x55911a2fa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fa700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55911a2f0d70 tx=0x7f05f400f790 comp rx=0 tx=0).stop 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 shutdown_connections 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a306bf0 0x55911a305e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f5ae0 0x55911a301740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f05e7fff640 0 client.15315 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fa700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 --2- 172.21.15.23:0/3034795715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2fa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 >> 172.21.15.23:0/3034795715 conn(0x55911a2eb4f0 msgr2=0x55911a2ea7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 shutdown_connections 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 1 -- 172.21.15.23:0/3034795715 wait complete. 2024-03-16T20:35:34.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.628+0000 7f0611d73740 20 client.15315 trim_cache size 0 max 16384 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 Processor -- start 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 -- start start 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f0970 0x55911a301700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a1dd490 con 0x55911a2f5ae0 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559119fa8270 con 0x55911a2eff80 2024-03-16T20:35:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0611d73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559119f576c0 con 0x55911a2f0970 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f060d652640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f060d652640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42012/0 (socket says 172.21.15.23:42012) 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f060d652640 1 -- 172.21.15.23:0/3114295226 learned_addr learned my addr 172.21.15.23:0/3114295226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f0607fff640 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f0970 0x55911a301700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.645+0000 7f060ce51640 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2f9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 -- 172.21.15.23:0/3114295226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f0970 msgr2=0x55911a301700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f0970 0x55911a301700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 -- 172.21.15.23:0/3114295226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 msgr2=0x55911a2f9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2f9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 -- 172.21.15.23:0/3114295226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55911a1f2a20 con 0x55911a2f5ae0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f060d652640 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0600009be0 tx=0x7f060002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbdfbe326d0a9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f05f27fc640 1 -- 172.21.15.23:0/3114295226 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060002cd60 con 0x55911a2f5ae0 2024-03-16T20:35:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f05f27fc640 1 -- 172.21.15.23:0/3114295226 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0600003e40 con 0x55911a2f5ae0 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f05f27fc640 1 -- 172.21.15.23:0/3114295226 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06000349a0 con 0x55911a2f5ae0 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 -- 172.21.15.23:0/3114295226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 msgr2=0x55911a2fa310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0600009be0 tx=0x7f060002e050 comp rx=0 tx=0).stop 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 -- 172.21.15.23:0/3114295226 shutdown_connections 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f5ae0 0x55911a2fa310 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2f0970 0x55911a301700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 --2- 172.21.15.23:0/3114295226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2eff80 0x55911a2f9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.646+0000 7f0611d73740 1 -- 172.21.15.23:0/3114295226 >> 172.21.15.23:0/3114295226 conn(0x55911a2eb4f0 msgr2=0x55911a2f1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- 172.21.15.23:0/3114295226 shutdown_connections 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- 172.21.15.23:0/3114295226 wait complete. 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- start start 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 Processor -- start 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2eff80 0x55911a2fe380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f5ae0 0x55911a361c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a3018e0 con 0x55911a2f0970 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a301a50 con 0x55911a2f5ae0 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55911a362190 con 0x55911a2eff80 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0611d73740 10 client.0 _ll_register_callbacks cb 0x7f05f4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0607fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0607fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42016/0 (socket says 172.21.15.23:42016) 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f0607fff640 1 -- 172.21.15.23:0/2926905476 learned_addr learned my addr 172.21.15.23:0/2926905476 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f060d652640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f5ae0 0x55911a361c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.647+0000 7f060ce51640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2eff80 0x55911a2fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 -- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2eff80 msgr2=0x55911a2fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2eff80 0x55911a2fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 -- 172.21.15.23:0/2926905476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f5ae0 msgr2=0x55911a361c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f5ae0 0x55911a361c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 -- 172.21.15.23:0/2926905476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0600049070 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0607fff640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55911a2fb6a0 tx=0x7f05f40108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706553ad571db430 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4017040 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f05f4000e40 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55911a306900 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f4005040 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 10 client.15321 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55911a218780 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55911a306900 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f05f4003a60 con 0x55911a2f0970 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 10 client.15321 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 10 client.15321 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:35:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 20 client.15321 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a367450 0x55911a3638d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 4 client.15321 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.648+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f05fc09ace0 con 0x55911a367450 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f060ce51640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a367450 0x55911a3638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f05e7fff640 10 client.15321 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f060ce51640 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a367450 0x55911a3638d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=975f09eee39092e6 server_cookie=a133c92818ea501b in_seq=0 out_seq=0 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f05e7fff640 1 -- 172.21.15.23:0/2926905476 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7f05fc09ace0 con 0x55911a367450 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f05e7fff640 10 client.15321 handle_command_reply: tid=0 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 client.15321 shutdown 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 msgr2=0x55911a2fe8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55911a2fb6a0 tx=0x7f05f40108b0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 shutdown_connections 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x55911a367450 0x55911a3638d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55911a2f5ae0 0x55911a361c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55911a2f0970 0x55911a2fe8c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f05e7fff640 0 client.15321 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 --2- 172.21.15.23:0/2926905476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55911a2eff80 0x55911a2fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 >> 172.21.15.23:0/2926905476 conn(0x55911a2eb4f0 msgr2=0x55911a2f1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 shutdown_connections 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.649+0000 7f0611d73740 1 -- 172.21.15.23:0/2926905476 wait complete. 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f0611d73740 20 client.15321 trim_cache size 0 max 16384 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-16T20:35:34.661 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24511, 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 104.439695484, 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 118.684902893 2024-03-16T20:35:34.662 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 msgr2=0x7f05f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 0x7f05f80a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0600004af0 tx=0x7f060002cad0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 msgr2=0x7f0608196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9f0 tx=0x7f05fc00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 shutdown_connections 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05f80a1a30 0x7f05f80a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0608101fc0 0x7f060819d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0608101600 0x7f0608196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 --2- 172.21.15.23:0/673585304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0608100410 0x7f0608196440 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:34.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 >> 172.21.15.23:0/673585304 conn(0x7f06080fb9a0 msgr2=0x7f0608104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:34.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 shutdown_connections 2024-03-16T20:35:34.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:34.650+0000 7f060f0b5640 1 -- 172.21.15.23:0/673585304 wait complete. 2024-03-16T20:35:34.665 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T20:35:34.666 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T20:35:34.675 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:35:34.675 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T20:35:34.920 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:36.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:36 smithi023 ceph-mon[31879]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-16T20:35:36.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:36 smithi031 ceph-mon[36281]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-16T20:35:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:36 smithi189 ceph-mon[36655]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.668+0000 7f3968a9d700 1 -- 172.21.15.23:0/3696582872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102b20 msgr2=0x7f39641093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.668+0000 7f3968a9d700 1 --2- 172.21.15.23:0/3696582872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102b20 0x7f39641093d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b0d0 tx=0x7f395c00b3e0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 -- 172.21.15.23:0/3696582872 shutdown_connections 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 --2- 172.21.15.23:0/3696582872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102b20 0x7f39641093d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 --2- 172.21.15.23:0/3696582872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964100f80 0x7f39641013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 --2- 172.21.15.23:0/3696582872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964102180 0x7f39641025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 -- 172.21.15.23:0/3696582872 >> 172.21.15.23:0/3696582872 conn(0x7f39640fc540 msgr2=0x7f39640fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 -- 172.21.15.23:0/3696582872 shutdown_connections 2024-03-16T20:35:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 -- 172.21.15.23:0/3696582872 wait complete. 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 Processor -- start 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.669+0000 7f3968a9d700 1 -- start start 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 0x7f3964198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964102b20 0x7f396419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396419fc20 con 0x7f3964102180 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396419fd60 con 0x7f3964100f80 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396419fef0 con 0x7f3964102b20 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42028/0 (socket says 172.21.15.23:42028) 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 -- 172.21.15.23:0/1546494755 learned_addr learned my addr 172.21.15.23:0/1546494755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f396259c700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 0x7f3964198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:36.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3962d9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964102b20 0x7f396419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 -- 172.21.15.23:0/1546494755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964102b20 msgr2=0x7f396419f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964102b20 0x7f396419f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 -- 172.21.15.23:0/1546494755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 msgr2=0x7f3964198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 0x7f3964198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 -- 172.21.15.23:0/1546494755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c009d00 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f396259c700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 0x7f3964198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3961d9b700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9d0 tx=0x7f395800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958009790 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39641a01d0 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f395800ee70 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958018680 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.670+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39641a0720 con 0x7f3964102180 2024-03-16T20:35:36.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.671+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3964068ca0 con 0x7f3964102180 2024-03-16T20:35:36.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.673+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f395800cbd0 con 0x7f3964102180 2024-03-16T20:35:36.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.673+0000 7f39537fe700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 0x7f394c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:36.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.673+0000 7f396259c700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 0x7f394c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:36.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.673+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3958014070 con 0x7f3964102180 2024-03-16T20:35:36.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.674+0000 7f396259c700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 0x7f394c0a3f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3964101fe0 tx=0x7f3954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:36.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.677+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f39580969d0 con 0x7f3964102180 2024-03-16T20:35:36.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.901+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39640fe670 con 0x7f394c0a1ac0 2024-03-16T20:35:36.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.902+0000 7f39537fe700 1 -- 172.21.15.23:0/1546494755 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f39640fe670 con 0x7f394c0a1ac0 2024-03-16T20:35:36.906 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-16T20:35:36.907 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-16T20:35:36.907 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 msgr2=0x7f394c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 0x7f394c0a3f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3964101fe0 tx=0x7f3954009260 comp rx=0 tx=0).stop 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 msgr2=0x7f39641985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9d0 tx=0x7f395800ece0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 shutdown_connections 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f394c0a1ac0 0x7f394c0a3f80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3964102b20 0x7f396419f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3964100f80 0x7f3964198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 --2- 172.21.15.23:0/1546494755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3964102180 0x7f39641985d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 >> 172.21.15.23:0/1546494755 conn(0x7f39640fc540 msgr2=0x7f39640fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 shutdown_connections 2024-03-16T20:35:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:36.904+0000 7f3968a9d700 1 -- 172.21.15.23:0/1546494755 wait complete. 2024-03-16T20:35:37.222 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:37 smithi023 ceph-mon[31879]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:37.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:37 smithi031 ceph-mon[36281]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:37 smithi189 ceph-mon[36655]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:37.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T20:35:37.910 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:38.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:38 smithi023 ceph-mon[31879]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:38.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:38 smithi031 ceph-mon[36281]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:38 smithi189 ceph-mon[36655]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:39.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.690+0000 7efd0cd9f700 1 -- 172.21.15.23:0/116680461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 msgr2=0x7efd081024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:39.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.690+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/116680461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd081024f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009990 tx=0x7efcfc009ca0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 -- 172.21.15.23:0/116680461 shutdown_connections 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/116680461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08102a30 0x7efd08109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/116680461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08100f10 0x7efd08101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/116680461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd081024f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 -- 172.21.15.23:0/116680461 >> 172.21.15.23:0/116680461 conn(0x7efd080fc4d0 msgr2=0x7efd080fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 -- 172.21.15.23:0/116680461 shutdown_connections 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 -- 172.21.15.23:0/116680461 wait complete. 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 Processor -- start 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.691+0000 7efd0cd9f700 1 -- start start 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08100f10 0x7efd08104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:39.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08102a30 0x7efd08103a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08104210 con 0x7efd08102070 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08104380 con 0x7efd08102a30 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd081a1fc0 con 0x7efd08100f10 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42046/0 (socket says 172.21.15.23:42046) 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 -- 172.21.15.23:0/1180564958 learned_addr learned my addr 172.21.15.23:0/1180564958 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0659c700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08100f10 0x7efd08104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd06d9d700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08102a30 0x7efd08103a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 -- 172.21.15.23:0/1180564958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08100f10 msgr2=0x7efd08104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08100f10 0x7efd08104f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 -- 172.21.15.23:0/1180564958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08102a30 msgr2=0x7efd08103a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08102a30 0x7efd08103a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 -- 172.21.15.23:0/1180564958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0095f0 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd05d9b700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00b5f0 tx=0x7efcfc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc01d070 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcfc00fb40 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd081a2240 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc017650 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.692+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd081a2650 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.693+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08068ca0 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.694+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efcfc00fcb0 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.695+0000 7efcf77fe700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 0x7efcf00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.695+0000 7efd0659c700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 0x7efcf00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.695+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7efcfc021e40 con 0x7efd08102070 2024-03-16T20:35:39.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.695+0000 7efd0659c700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 0x7efcf00a3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009c60 tx=0x7efcf8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:39.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.699+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efcfc097fc0 con 0x7efd08102070 2024-03-16T20:35:39.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.915+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efd0806ae10 con 0x7efcf00a1ac0 2024-03-16T20:35:39.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.929+0000 7efcf77fe700 1 -- 172.21.15.23:0/1180564958 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7efd0806ae10 con 0x7efcf00a1ac0 2024-03-16T20:35:39.934 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T20:35:39.934 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (118s) 20s ago 118s 19.0M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 313baabf8379 2024-03-16T20:35:39.934 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi031 running (115s) 48s ago 115s 20.0M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 3535f825bb2d 2024-03-16T20:35:39.934 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi189 running (2m) 48s ago 2m 17.5M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 4217464e164f 2024-03-16T20:35:39.934 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (2m) 20s ago 2m 13.7M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 070cec807325 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi031 running (2m) 48s ago 2m 17.8M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 8954f2554da9 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi189 running (2m) 48s ago 2m 16.5M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 9a4b272cb36c 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (2m) 20s ago 2m 18.9M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 7fbb4133a62b 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi031 running (2m) 48s ago 2m 20.0M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 85cc5cf23ed9 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi189 running (2m) 48s ago 2m 26.1M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 73bfe9c74303 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (2m) 20s ago 2m 14.0M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba a86939ed9d53 2024-03-16T20:35:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi031 running (2m) 48s ago 2m 17.4M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 7842f3b7956c 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi189 running (2m) 48s ago 2m 18.9M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 2f47d882d900 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (9m) 20s ago 9m 514M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 3e3fa79e76d0 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi031 *:8443,8765 running (7m) 48s ago 7m 453M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba a18bd07d2401 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi189 *:8443,8765 running (7m) 48s ago 7m 453M - 19.0.0-1681-gaba3c1d2 80ec0d3daaba 89a5d412cebe 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (9m) 20s ago 9m 53.8M 2048M 19.0.0-1681-gaba3c1d2 80ec0d3daaba bb9eecf41469 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi031 running (7m) 48s ago 7m 37.9M 2048M 19.0.0-1681-gaba3c1d2 80ec0d3daaba cc4ab3c979c4 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi189 running (8m) 48s ago 8m 39.5M 2048M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 11bcfd600bc1 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (7m) 20s ago 7m 57.4M 4272M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 0d1a9eaf3c72 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi031 running (7m) 48s ago 7m 57.2M 4271M 19.0.0-1681-gaba3c1d2 80ec0d3daaba c0a73288a976 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi189 running (6m) 48s ago 6m 58.7M 4270M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 2a2abcaa72c9 2024-03-16T20:35:39.936 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (6m) 20s ago 6m 55.5M 4272M 19.0.0-1681-gaba3c1d2 80ec0d3daaba a6c2e637c505 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi031 running (5m) 48s ago 5m 42.9M 4271M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 6b03886c1126 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi189 running (5m) 48s ago 5m 45.2M 4270M 19.0.0-1681-gaba3c1d2 80ec0d3daaba cdc4ee518188 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (4m) 20s ago 4m 42.3M 4272M 19.0.0-1681-gaba3c1d2 80ec0d3daaba e5e25ad3dec2 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi031 running (4m) 48s ago 4m 55.9M 4271M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 71b5267411aa 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi189 running (4m) 48s ago 4m 54.7M 4270M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 945e4cfd0d88 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (3m) 20s ago 3m 53.5M 4272M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 0a18d1091c2a 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi031 running (3m) 48s ago 3m 43.6M 4271M 19.0.0-1681-gaba3c1d2 80ec0d3daaba fb1cc96208b5 2024-03-16T20:35:39.937 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi189 running (2m) 48s ago 2m 51.8M 4270M 19.0.0-1681-gaba3c1d2 80ec0d3daaba 0c312cb6bdcb 2024-03-16T20:35:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 msgr2=0x7efcf00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 0x7efcf00a3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009c60 tx=0x7efcf8009260 comp rx=0 tx=0).stop 2024-03-16T20:35:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 msgr2=0x7efd08103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00b5f0 tx=0x7efcfc009580 comp rx=0 tx=0).stop 2024-03-16T20:35:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 shutdown_connections 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7efcf00a1ac0 0x7efcf00a3f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd08102a30 0x7efd08103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08100f10 0x7efd08104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 --2- 172.21.15.23:0/1180564958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd08102070 0x7efd08103550 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 >> 172.21.15.23:0/1180564958 conn(0x7efd080fc4d0 msgr2=0x7efd0806a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 shutdown_connections 2024-03-16T20:35:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:39.931+0000 7efd0cd9f700 1 -- 172.21.15.23:0/1180564958 wait complete. 2024-03-16T20:35:40.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:40 smithi023 ceph-mon[31879]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:40.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:40 smithi023 ceph-mon[31879]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:40 smithi031 ceph-mon[36281]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:40 smithi031 ceph-mon[36281]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:40 smithi189 ceph-mon[36655]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:40.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:40 smithi189 ceph-mon[36655]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:40.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T20:35:40.949 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:42.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:42 smithi023 ceph-mon[31879]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:42.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:42 smithi031 ceph-mon[36281]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:42.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:42 smithi189 ceph-mon[36655]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:42.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 -- 172.21.15.23:0/502141382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 msgr2=0x7f0fcc104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:42.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 --2- 172.21.15.23:0/502141382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc104e90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400b0a0 tx=0x7f0fc400b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 -- 172.21.15.23:0/502141382 shutdown_connections 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 --2- 172.21.15.23:0/502141382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc104e90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 --2- 172.21.15.23:0/502141382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc107d00 0x7f0fcc10a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 --2- 172.21.15.23:0/502141382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc1053d0 0x7f0fcc1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 -- 172.21.15.23:0/502141382 >> 172.21.15.23:0/502141382 conn(0x7f0fcc0fc4d0 msgr2=0x7f0fcc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 -- 172.21.15.23:0/502141382 shutdown_connections 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 -- 172.21.15.23:0/502141382 wait complete. 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.711+0000 7f0fd3356700 1 Processor -- start 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 -- start start 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc1053d0 0x7f0fcc100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc107d00 0x7f0fcc100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc101710 con 0x7f0fcc102aa0 2024-03-16T20:35:42.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc101880 con 0x7f0fcc1053d0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc1a2170 con 0x7f0fcc107d00 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55266/0 (socket says 172.21.15.23:55266) 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 -- 172.21.15.23:0/477511492 learned_addr learned my addr 172.21.15.23:0/477511492 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd08f1700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc1053d0 0x7f0fcc100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd18f3700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc107d00 0x7f0fcc100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 -- 172.21.15.23:0/477511492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc107d00 msgr2=0x7f0fcc100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc107d00 0x7f0fcc100f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 -- 172.21.15.23:0/477511492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc1053d0 msgr2=0x7f0fcc100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc1053d0 0x7f0fcc100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 -- 172.21.15.23:0/477511492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4009d00 con 0x7f0fcc102aa0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd10f2700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009610 tx=0x7f0fc40079b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400e040 con 0x7f0fcc102aa0 2024-03-16T20:35:42.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc4004c50 con 0x7f0fcc102aa0 2024-03-16T20:35:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.712+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc1a23f0 con 0x7f0fcc102aa0 2024-03-16T20:35:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.713+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40052c0 con 0x7f0fcc102aa0 2024-03-16T20:35:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.713+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc1a28b0 con 0x7f0fcc102aa0 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.714+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc0fe050 con 0x7f0fcc102aa0 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.715+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0fc4005420 con 0x7f0fcc102aa0 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.715+0000 7f0fca7fc700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 0x7f0fb40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.716+0000 7f0fd08f1700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 0x7f0fb40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.716+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0fc40ceea0 con 0x7f0fcc102aa0 2024-03-16T20:35:42.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.716+0000 7f0fd08f1700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 0x7f0fb40a4000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009c60 tx=0x7f0fc0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:42.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.719+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0fc4097e00 con 0x7f0fcc102aa0 2024-03-16T20:35:42.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.932+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fcc02da30 con 0x7f0fb40a1b40 2024-03-16T20:35:42.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.938+0000 7f0fca7fc700 1 -- 172.21.15.23:0/477511492 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f0fcc02da30 con 0x7f0fb40a1b40 2024-03-16T20:35:42.943 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T20:35:42.943 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 51s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi031=b;smithi031=e;smithi031=h;smithi031=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-03-16T20:35:42.943 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 51s ago 7m smithi023=x;smithi031=y;smithi189=z;count:3 2024-03-16T20:35:42.943 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 51s ago 8m smithi023:172.21.15.23=a;smithi031:172.21.15.31=b;smithi189:172.21.15.189=c;count:3 2024-03-16T20:35:42.944 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 51s ago - 2024-03-16T20:35:42.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 msgr2=0x7f0fb40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 0x7f0fb40a4000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009c60 tx=0x7f0fc0009260 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 msgr2=0x7f0fcc102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009610 tx=0x7f0fc40079b0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 shutdown_connections 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0fb40a1b40 0x7f0fb40a4000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fcc102aa0 0x7f0fcc102400 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fcc107d00 0x7f0fcc100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 --2- 172.21.15.23:0/477511492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fcc1053d0 0x7f0fcc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.940+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 >> 172.21.15.23:0/477511492 conn(0x7f0fcc0fc4d0 msgr2=0x7f0fcc103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.941+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 shutdown_connections 2024-03-16T20:35:42.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:42.941+0000 7f0fd3356700 1 -- 172.21.15.23:0/477511492 wait complete. 2024-03-16T20:35:43.287 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:43 smithi023 ceph-mon[31879]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:43.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:43 smithi031 ceph-mon[36281]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:43.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:43 smithi189 ceph-mon[36655]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:43.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T20:35:43.968 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:44.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:44 smithi023 ceph-mon[31879]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:44.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:44 smithi031 ceph-mon[36281]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:44.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:44 smithi189 ceph-mon[36655]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:45.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 -- 172.21.15.23:0/2683418453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102a20 msgr2=0x7f6060109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 --2- 172.21.15.23:0/2683418453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102a20 0x7f6060109340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f605000b0a0 tx=0x7f605000b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 -- 172.21.15.23:0/2683418453 shutdown_connections 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 --2- 172.21.15.23:0/2683418453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102a20 0x7f6060109340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 --2- 172.21.15.23:0/2683418453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060100f00 0x7f6060101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 --2- 172.21.15.23:0/2683418453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060102060 0x7f60601024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 -- 172.21.15.23:0/2683418453 >> 172.21.15.23:0/2683418453 conn(0x7f60600fc4a0 msgr2=0x7f60600fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 -- 172.21.15.23:0/2683418453 shutdown_connections 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.702+0000 7f6068506700 1 -- 172.21.15.23:0/2683418453 wait complete. 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 Processor -- start 2024-03-16T20:35:45.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 -- start start 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 0x7f6060198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060102a20 0x7f606019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606019fc70 con 0x7f6060102060 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606019fdb0 con 0x7f6060100f00 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6068506700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606019ff40 con 0x7f6060102a20 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6065aa1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 0x7f6060198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f60662a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f60662a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:34472/0 (socket says 172.21.15.23:34472) 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f60662a2700 1 -- 172.21.15.23:0/1446281868 learned_addr learned my addr 172.21.15.23:0/1446281868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.703+0000 7f6066aa3700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060102a20 0x7f606019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 -- 172.21.15.23:0/1446281868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060102a20 msgr2=0x7f606019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060102a20 0x7f606019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 -- 172.21.15.23:0/1446281868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 msgr2=0x7f60601980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 -- 172.21.15.23:0/1446281868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050009d00 con 0x7f6060102060 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f60662a2700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6065aa1700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 0x7f6060198620 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f605400ea20 tx=0x7f605400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605400cbd0 con 0x7f6060102060 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60601a0220 con 0x7f6060102060 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f605400ee60 con 0x7f6060102060 2024-03-16T20:35:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054018680 con 0x7f6060102060 2024-03-16T20:35:45.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.704+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60601a0720 con 0x7f6060102060 2024-03-16T20:35:45.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.705+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60601081e0 con 0x7f6060102060 2024-03-16T20:35:45.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.706+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6054010430 con 0x7f6060102060 2024-03-16T20:35:45.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.707+0000 7f605f7fe700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 0x7f60480a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:45.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.707+0000 7f60662a2700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 0x7f60480a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:45.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.707+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f6054014070 con 0x7f6060102060 2024-03-16T20:35:45.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.707+0000 7f60662a2700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 0x7f60480a3f80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6058011550 tx=0x7f6058009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.711+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6054053280 con 0x7f6060102060 2024-03-16T20:35:45.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.923+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f606002da30 con 0x7f60480a1ac0 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.924+0000 7f605f7fe700 1 -- 172.21.15.23:0/1446281868 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f606002da30 con 0x7f60480a1ac0 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:smithi031 172.21.15.31 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:smithi189 172.21.15.189 2024-03-16T20:35:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 msgr2=0x7f60480a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 0x7f60480a3f80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6058011550 tx=0x7f6058009260 comp rx=0 tx=0).stop 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 msgr2=0x7f6060198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 0x7f6060198620 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f605400ea20 tx=0x7f605400ed30 comp rx=0 tx=0).stop 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 shutdown_connections 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f60480a1ac0 0x7f60480a3f80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6060102a20 0x7f606019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6060100f00 0x7f60601980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 --2- 172.21.15.23:0/1446281868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6060102060 0x7f6060198620 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:45.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 >> 172.21.15.23:0/1446281868 conn(0x7f60600fc4a0 msgr2=0x7f6060105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:45.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 shutdown_connections 2024-03-16T20:35:45.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:45.926+0000 7f6068506700 1 -- 172.21.15.23:0/1446281868 wait complete. 2024-03-16T20:35:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:46 smithi023 ceph-mon[31879]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:46.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:46 smithi023 ceph-mon[31879]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:46 smithi031 ceph-mon[36281]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:46 smithi031 ceph-mon[36281]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:46.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:46 smithi189 ceph-mon[36655]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:46.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:46 smithi189 ceph-mon[36655]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:46.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T20:35:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:48.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:48 smithi023 ceph-mon[31879]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:48.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:48 smithi031 ceph-mon[36281]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:48.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:48 smithi189 ceph-mon[36655]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:48.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 -- 172.21.15.23:0/1302492818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8102010 msgr2=0x7fcdf8102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/1302492818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8102010 0x7fcdf8102430 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0099c0 tx=0x7fcdec009cd0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 -- 172.21.15.23:0/1302492818 shutdown_connections 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/1302492818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8102010 0x7fcdf8102430 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/1302492818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8103bb0 0x7fcdf810a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/1302492818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103210 0x7fcdf8103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 -- 172.21.15.23:0/1302492818 >> 172.21.15.23:0/1302492818 conn(0x7fcdf80fd5f0 msgr2=0x7fcdf80ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 -- 172.21.15.23:0/1302492818 shutdown_connections 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.723+0000 7fcdfd19a700 1 -- 172.21.15.23:0/1302492818 wait complete. 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 Processor -- start 2024-03-16T20:35:48.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 -- start start 2024-03-16T20:35:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8102010 0x7fcdf8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 0x7fcdf819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf819fb60 con 0x7fcdf8103210 2024-03-16T20:35:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf819fca0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdfd19a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf819fe30 con 0x7fcdf8102010 2024-03-16T20:35:48.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdf659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:48.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdf659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55312/0 (socket says 172.21.15.23:55312) 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdf759e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 0x7fcdf819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdf659c700 1 -- 172.21.15.23:0/3545163283 learned_addr learned my addr 172.21.15.23:0/3545163283 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.724+0000 7fcdf6d9d700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8102010 0x7fcdf8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 -- 172.21.15.23:0/3545163283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8102010 msgr2=0x7fcdf8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8102010 0x7fcdf8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 -- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 msgr2=0x7fcdf8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 -- 172.21.15.23:0/3545163283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0095f0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf659c700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdf759e700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 0x7fcdf819f590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000efc0 tx=0x7fcdf000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0018070 con 0x7fcdf8103bb0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf81a00b0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdf000f040 con 0x7fcdf8103bb0 2024-03-16T20:35:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000ce00 con 0x7fcdf8103bb0 2024-03-16T20:35:48.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.725+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf81a05b0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.726+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf81093d0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.727+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcdf0004ad0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.728+0000 7fcddffff700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 0x7fcde00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.728+0000 7fcdf6d9d700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 0x7fcde00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.728+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fcdf00cdbd0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.728+0000 7fcdf6d9d700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 0x7fcde00a3f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0036c0 tx=0x7fcdec005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:48.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.736+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcdf0096ad0 con 0x7fcdf8103bb0 2024-03-16T20:35:48.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.953+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdf802da30 con 0x7fcde00a1ac0 2024-03-16T20:35:48.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.955+0000 7fcddffff700 1 -- 172.21.15.23:0/3545163283 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fcdf802da30 con 0x7fcde00a1ac0 2024-03-16T20:35:48.960 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T20:35:48.960 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-03-16T20:35:48.960 INFO:teuthology.orchestra.run.smithi023.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-16T20:35:48.960 INFO:teuthology.orchestra.run.smithi023.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 msgr2=0x7fcde00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 0x7fcde00a3f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0036c0 tx=0x7fcdec005ad0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 msgr2=0x7fcdf819f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 0x7fcdf819f590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000efc0 tx=0x7fcdf000c3a0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 shutdown_connections 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fcde00a1ac0 0x7fcde00a3f80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf8103bb0 0x7fcdf819f590 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdf8102010 0x7fcdf8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 --2- 172.21.15.23:0/3545163283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf8103210 0x7fcdf8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:48.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 >> 172.21.15.23:0/3545163283 conn(0x7fcdf80fd5f0 msgr2=0x7fcdf8106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:48.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 shutdown_connections 2024-03-16T20:35:48.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:48.957+0000 7fcdfd19a700 1 -- 172.21.15.23:0/3545163283 wait complete. 2024-03-16T20:35:49.680 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T20:35:49.693 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:35:49.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-16T20:35:49.947 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:50.213 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:49 smithi023 ceph-mon[31879]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:50.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:49 smithi189 ceph-mon[36655]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:50.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:49 smithi031 ceph-mon[36281]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:35:51.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:50 smithi023 ceph-mon[31879]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:51.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:50 smithi189 ceph-mon[36655]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:50 smithi031 ceph-mon[36281]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:35:51.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- 172.21.15.23:0/1836025021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324103610 msgr2=0x7f7324103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:51.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 --2- 172.21.15.23:0/1836025021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324103610 0x7f7324103a30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f731c009990 tx=0x7f731c009ca0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- 172.21.15.23:0/1836025021 shutdown_connections 2024-03-16T20:35:51.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 --2- 172.21.15.23:0/1836025021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324103610 0x7f7324103a30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 --2- 172.21.15.23:0/1836025021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324100a30 0x7f732410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 --2- 172.21.15.23:0/1836025021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100090 0x7f73241004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- 172.21.15.23:0/1836025021 >> 172.21.15.23:0/1836025021 conn(0x7f73240fbc30 msgr2=0x7f73240fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- 172.21.15.23:0/1836025021 shutdown_connections 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- 172.21.15.23:0/1836025021 wait complete. 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 Processor -- start 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.586+0000 7f732aee3700 1 -- start start 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100a30 0x7f7324101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324103610 0x7f73241018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7324102020 con 0x7f7324100090 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7324102190 con 0x7f7324103610 2024-03-16T20:35:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73241a2020 con 0x7f7324100a30 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38612/0 (socket says 172.21.15.23:38612) 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 -- 172.21.15.23:0/4283788419 learned_addr learned my addr 172.21.15.23:0/4283788419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7329480700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324103610 0x7f73241018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7323fff700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100a30 0x7f7324101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 -- 172.21.15.23:0/4283788419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100a30 msgr2=0x7f7324101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100a30 0x7f7324101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 -- 172.21.15.23:0/4283788419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324103610 msgr2=0x7f73241018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324103610 0x7f73241018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 -- 172.21.15.23:0/4283788419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0095f0 con 0x7f7324100090 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7328c7f700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f731c000c00 tx=0x7f731c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:51.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c01d070 con 0x7f7324100090 2024-03-16T20:35:51.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73241a22a0 con 0x7f7324100090 2024-03-16T20:35:51.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f731c021410 con 0x7f7324100090 2024-03-16T20:35:51.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.587+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c017650 con 0x7f7324100090 2024-03-16T20:35:51.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.588+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73241a2740 con 0x7f7324100090 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.588+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7324109570 con 0x7f7324100090 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.590+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f731c021580 con 0x7f7324100090 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.590+0000 7f7321ffb700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 0x7f73100a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.590+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f731c0cea30 con 0x7f7324100090 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.590+0000 7f7323fff700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 0x7f73100a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.591+0000 7f7323fff700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 0x7f73100a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f731400f540 tx=0x7f731400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.594+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f731c097990 con 0x7f7324100090 2024-03-16T20:35:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.843+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f732410d970 con 0x7f7324100090 2024-03-16T20:35:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.843+0000 7f7321ffb700 1 -- 172.21.15.23:0/4283788419 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+3031 (secure 0 0 0) 0x7f731c030bf0 con 0x7f7324100090 2024-03-16T20:35:51.848 INFO:teuthology.orchestra.run.smithi023.stdout:e28 2024-03-16T20:35:51.848 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T20:35:51.848 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 28 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-16T20:33:49.239658+0000 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-16T20:34:10.946679+0000 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2024-03-16T20:35:51.849 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 5 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1,2,3,4 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24511,1=24497,2=24493,3=24418,4=24416} 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3,4] 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2024-03-16T20:35:51.850 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:qdb_cluster leader: 24511 members: 24497,24511,24418,24493,24416 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{0:24511} state up:active seq 5 addr [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{0:24481} state up:standby-replay seq 1 addr [v2:172.21.15.189:6836/3204120502,v1:172.21.15.189:6837/3204120502] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{1:24497} state up:active seq 7 addr [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{1:24469} state up:standby-replay seq 1 addr [v2:172.21.15.189:6834/1658699703,v1:172.21.15.189:6835/1658699703] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{2:24493} state up:active seq 10 addr [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{2:24449} state up:standby-replay seq 1 addr [v2:172.21.15.31:6834/354681253,v1:172.21.15.31:6835/354681253] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{3:24418} state up:active seq 15 addr [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{3:14655} state up:standby-replay seq 1 addr [v2:172.21.15.23:6840/4113023748,v1:172.21.15.23:6841/4113023748] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{4:24416} state up:active seq 15 addr [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{4:14643} state up:standby-replay seq 1 addr [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.851 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:51.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:51.852 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2024-03-16T20:35:51.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:51.852 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14562} state up:standby seq 1 addr [v2:172.21.15.23:6834/2314600527,v1:172.21.15.23:6835/2314600527] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.852 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.23:6836/1010201356,v1:172.21.15.23:6837/1010201356] compat {c=[1],r=[1],i=[fff]}] 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.846+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 msgr2=0x7f73100a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.846+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 0x7f73100a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f731400f540 tx=0x7f731400b040 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.846+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 msgr2=0x7f7324102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.846+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f731c000c00 tx=0x7f731c0047b0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 shutdown_connections 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f73100a1ac0 0x7f73100a3f80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7324100090 0x7f7324102d10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7324103610 0x7f73241018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 --2- 172.21.15.23:0/4283788419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7324100a30 0x7f7324101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 >> 172.21.15.23:0/4283788419 conn(0x7f73240fbc30 msgr2=0x7f73241066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 shutdown_connections 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:51.847+0000 7f732aee3700 1 -- 172.21.15.23:0/4283788419 wait complete. 2024-03-16T20:35:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:52.217 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:51 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4283788419' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T20:35:52.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:51 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4283788419' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T20:35:52.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:51 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4283788419' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-16T20:35:52.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-16T20:35:52.816 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:35:53.188 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:52 smithi023 ceph-mon[31879]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:53.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:52 smithi031 ceph-mon[36281]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:53.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:52 smithi189 ceph-mon[36655]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:35:54.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:54 smithi023 ceph-mon[31879]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:54 smithi031 ceph-mon[36281]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:54 smithi189 ceph-mon[36655]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- 172.21.15.23:0/1860704184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a0102ab0 msgr2=0x7f47a01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 --2- 172.21.15.23:0/1860704184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a0102ab0 0x7f47a01093d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b0a0 tx=0x7f479c00b3b0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- 172.21.15.23:0/1860704184 shutdown_connections 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 --2- 172.21.15.23:0/1860704184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a0102ab0 0x7f47a01093d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 --2- 172.21.15.23:0/1860704184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0100f10 0x7f47a0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 --2- 172.21.15.23:0/1860704184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a01020f0 0x7f47a0102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- 172.21.15.23:0/1860704184 >> 172.21.15.23:0/1860704184 conn(0x7f47a00fc500 msgr2=0x7f47a00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- 172.21.15.23:0/1860704184 shutdown_connections 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- 172.21.15.23:0/1860704184 wait complete. 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 Processor -- start 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.507+0000 7f47a72cd700 1 -- start start 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a0100f10 0x7f47a0104f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0102ab0 0x7f47a0103aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0104220 con 0x7f47a01020f0 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0104390 con 0x7f47a0100f10 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a72cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01a20f0 con 0x7f47a0102ab0 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a5069700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a0100f10 0x7f47a0104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38632/0 (socket says 172.21.15.23:38632) 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 -- 172.21.15.23:0/368785513 learned_addr learned my addr 172.21.15.23:0/368785513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a586a700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0102ab0 0x7f47a0103aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 -- 172.21.15.23:0/368785513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0102ab0 msgr2=0x7f47a0103aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0102ab0 0x7f47a0103aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 -- 172.21.15.23:0/368785513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a0100f10 msgr2=0x7f47a0104f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a0100f10 0x7f47a0104f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 -- 172.21.15.23:0/368785513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c009d00 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47a4868700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f479000e9a0 tx=0x7f479000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790009790 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4790010430 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.508+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790018680 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.509+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a01a23d0 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.509+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a01a2880 con 0x7f47a01020f0 2024-03-16T20:35:54.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.509+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0068ca0 con 0x7f47a01020f0 2024-03-16T20:35:54.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.511+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4790010910 con 0x7f47a01020f0 2024-03-16T20:35:54.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.511+0000 7f47967fc700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 0x7f478c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:54.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.511+0000 7f47a5069700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 0x7f478c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:54.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.511+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4790014070 con 0x7f47a01020f0 2024-03-16T20:35:54.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.512+0000 7f47a5069700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 0x7f478c0a3f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f47a0100c40 tx=0x7f4798009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:54.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.515+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4790096d10 con 0x7f47a01020f0 2024-03-16T20:35:54.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.721+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f47a010c8b0 con 0x7f47a01020f0 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.722+0000 7f47967fc700 1 -- 172.21.15.23:0/368785513 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v92) v1 ==== 56+0+5561 (secure 0 0 0) 0x7f47900105a0 con 0x7f47a01020f0 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 92 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-16T20:25:46.502894+0000 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-16T20:34:51.409333+0000 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 27 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-16T20:35:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release squid 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/3821836785,v1:172.21.15.23:6803/3821836785] [v2:172.21.15.23:6804/3821836785,v1:172.21.15.23:6805/3821836785] exists,up 96865589-182e-46bf-b061-3b6f0521156b 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6800/639380907,v1:172.21.15.31:6801/639380907] [v2:172.21.15.31:6802/639380907,v1:172.21.15.31:6803/639380907] exists,up 8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3 2024-03-16T20:35:54.728 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/3710574436,v1:172.21.15.189:6801/3710574436] [v2:172.21.15.189:6802/3710574436,v1:172.21.15.189:6803/3710574436] exists,up 07b0e4ef-221b-4cfa-b176-8ff89dda1c90 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/1075234291,v1:172.21.15.23:6811/1075234291] [v2:172.21.15.23:6812/1075234291,v1:172.21.15.23:6813/1075234291] exists,up cc0249a5-38ab-46a3-957f-f13367ec3060 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6808/2047837401,v1:172.21.15.31:6809/2047837401] [v2:172.21.15.31:6810/2047837401,v1:172.21.15.31:6811/2047837401] exists,up 20ff34b3-508e-49ca-a5aa-61c2c52551b1 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 38 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/2617907376,v1:172.21.15.189:6809/2617907376] [v2:172.21.15.189:6810/2617907376,v1:172.21.15.189:6811/2617907376] exists,up 7b28071d-a593-49b1-9561-614bf17df712 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 46 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/103652673,v1:172.21.15.23:6819/103652673] [v2:172.21.15.23:6820/103652673,v1:172.21.15.23:6821/103652673] exists,up e7698cf3-f3a6-4e39-991a-b101bc81d7ec 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 51 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6816/2033067958,v1:172.21.15.31:6817/2033067958] [v2:172.21.15.31:6818/2033067958,v1:172.21.15.31:6819/2033067958] exists,up 82df9886-90ce-4a0a-8bdc-09b78caf46af 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 60 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/3435219550,v1:172.21.15.189:6817/3435219550] [v2:172.21.15.189:6818/3435219550,v1:172.21.15.189:6819/3435219550] exists,up 49cd9654-b197-40a7-ab09-f7d720b898ff 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/1125641919,v1:172.21.15.23:6827/1125641919] [v2:172.21.15.23:6828/1125641919,v1:172.21.15.23:6829/1125641919] exists,up 38b7a821-6c63-4636-84e2-cfa25c440244 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6824/3304713388,v1:172.21.15.31:6825/3304713388] [v2:172.21.15.31:6826/3304713388,v1:172.21.15.31:6827/3304713388] exists,up 6615be9c-9a48-448d-af63-51c8ab219057 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/702443898,v1:172.21.15.189:6825/702443898] [v2:172.21.15.189:6826/702443898,v1:172.21.15.189:6827/702443898] exists,up 7877d354-a2b5-4c0e-844d-b56717ad5a9b 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.a [2,11] 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.b [8,11] 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.2 [3,9] 2024-03-16T20:35:54.729 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.24 [4,9] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.2c [4,9] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.b [9,10] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.e [5,8] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1b [9,8] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1f [9,8] 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/2899821830 expires 2024-03-17T20:26:49.856928+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1269138528 expires 2024-03-17T20:26:23.333931+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/4262074056 expires 2024-03-17T20:26:49.856928+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/3890527981 expires 2024-03-17T20:26:23.333931+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2140563160 expires 2024-03-17T20:26:49.856928+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/4012523374 expires 2024-03-17T20:26:49.856928+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1525116937 expires 2024-03-17T20:26:23.333931+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/3890527981 expires 2024-03-17T20:26:23.333931+0000 2024-03-16T20:35:54.730 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3223781845 expires 2024-03-17T20:26:23.333931+0000 2024-03-16T20:35:54.731 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/2899821830 expires 2024-03-17T20:26:49.856928+0000 2024-03-16T20:35:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 msgr2=0x7f478c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 0x7f478c0a3f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f47a0100c40 tx=0x7f4798009260 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 msgr2=0x7f47a0103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f479000e9a0 tx=0x7f479000ecb0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 shutdown_connections 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f478c0a1ac0 0x7f478c0a3f80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47a0102ab0 0x7f47a0103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47a0100f10 0x7f47a0104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 --2- 172.21.15.23:0/368785513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a01020f0 0x7f47a0103560 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 >> 172.21.15.23:0/368785513 conn(0x7f47a00fc500 msgr2=0x7f47a00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 shutdown_connections 2024-03-16T20:35:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:54.725+0000 7f47a72cd700 1 -- 172.21.15.23:0/368785513 wait complete. 2024-03-16T20:35:55.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:55 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/368785513' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T20:35:55.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:55 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/368785513' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T20:35:55.462 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-16T20:35:55.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:55.474 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:55 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/368785513' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-16T20:35:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 -- 172.21.15.23:0/4234953537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac101520 msgr2=0x7f64ac101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 --2- 172.21.15.23:0/4234953537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac101520 0x7f64ac101940 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 -- 172.21.15.23:0/4234953537 shutdown_connections 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 --2- 172.21.15.23:0/4234953537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac1030d0 0x7f64ac109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 --2- 172.21.15.23:0/4234953537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 0x7f64ac102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 --2- 172.21.15.23:0/4234953537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac101520 0x7f64ac101940 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 -- 172.21.15.23:0/4234953537 >> 172.21.15.23:0/4234953537 conn(0x7f64ac0fcad0 msgr2=0x7f64ac0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 -- 172.21.15.23:0/4234953537 shutdown_connections 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.575+0000 7f64b0977640 1 -- 172.21.15.23:0/4234953537 wait complete. 2024-03-16T20:35:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 Processor -- start 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 -- start start 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac101520 0x7f64ac1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 0x7f64ac197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19f0e0 con 0x7f64ac1030d0 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19f250 con 0x7f64ac101520 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64b0977640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19f410 con 0x7f64ac102710 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38638/0 (socket says 172.21.15.23:38638) 2024-03-16T20:35:55.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 -- 172.21.15.23:0/2889043726 learned_addr learned my addr 172.21.15.23:0/2889043726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64a9d9b640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 0x7f64ac197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aa59c640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac101520 0x7f64ac1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 -- 172.21.15.23:0/2889043726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 msgr2=0x7f64ac197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 0x7f64ac197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 -- 172.21.15.23:0/2889043726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac101520 msgr2=0x7f64ac1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac101520 0x7f64ac1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.576+0000 7f64aad9d640 1 -- 172.21.15.23:0/2889043726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c009560 con 0x7f64ac1030d0 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64aad9d640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f64a400eb10 tx=0x7f64a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4004b20 con 0x7f64ac1030d0 2024-03-16T20:35:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a4004c80 con 0x7f64ac1030d0 2024-03-16T20:35:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac19f6f0 con 0x7f64ac1030d0 2024-03-16T20:35:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4010650 con 0x7f64ac1030d0 2024-03-16T20:35:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.577+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac19fbb0 con 0x7f64ac1030d0 2024-03-16T20:35:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.578+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac108910 con 0x7f64ac1030d0 2024-03-16T20:35:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.579+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f64a4005020 con 0x7f64ac1030d0 2024-03-16T20:35:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.579+0000 7f64a37fe640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.579+0000 7f64aa59c640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.579+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f64a4014070 con 0x7f64ac1030d0 2024-03-16T20:35:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.580+0000 7f64aa59c640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 0x7f64980a3ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f649c005ab0 tx=0x7f649c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.584+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f64a4096bd0 con 0x7f64ac1030d0 2024-03-16T20:35:55.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.767+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64ac108380 con 0x7f64ac1030d0 2024-03-16T20:35:55.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.769+0000 7f64a37fe640 1 -- 172.21.15.23:0/2889043726 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f64a409a290 con 0x7f64ac1030d0 2024-03-16T20:35:55.774 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:55.774 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:55.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.770+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 msgr2=0x7f64980a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.770+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 0x7f64980a3ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f649c005ab0 tx=0x7f649c005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.770+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 msgr2=0x7f64ac19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.770+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f64a400eb10 tx=0x7f64a400c2d0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 shutdown_connections 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f64980a1a30 0x7f64980a3ef0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64ac1030d0 0x7f64ac19eba0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64ac102710 0x7f64ac197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 --2- 172.21.15.23:0/2889043726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64ac101520 0x7f64ac1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 >> 172.21.15.23:0/2889043726 conn(0x7f64ac0fcad0 msgr2=0x7f64ac105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:55.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 shutdown_connections 2024-03-16T20:35:55.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.771+0000 7f64b0977640 1 -- 172.21.15.23:0/2889043726 wait complete. 2024-03-16T20:35:55.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:35:55.787 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 -- 172.21.15.23:0/3265881396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924100400 msgr2=0x7f7924100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 --2- 172.21.15.23:0/3265881396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924100400 0x7f7924100820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7920009900 tx=0x7f792002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 -- 172.21.15.23:0/3265881396 shutdown_connections 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 --2- 172.21.15.23:0/3265881396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924101fb0 0x7f7924108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 --2- 172.21.15.23:0/3265881396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 0x7f7924101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 --2- 172.21.15.23:0/3265881396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924100400 0x7f7924100820 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 -- 172.21.15.23:0/3265881396 >> 172.21.15.23:0/3265881396 conn(0x7f79240fb990 msgr2=0x7f79240fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 -- 172.21.15.23:0/3265881396 shutdown_connections 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.888+0000 7f792bd0d640 1 -- 172.21.15.23:0/3265881396 wait complete. 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 Processor -- start 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 -- start start 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 0x7f79241029d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924101fb0 0x7f7924102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7924103670 con 0x7f7924100400 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79241037e0 con 0x7f7924101fb0 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792bd0d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79241a16b0 con 0x7f79241015f0 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38652/0 (socket says 172.21.15.23:38652) 2024-03-16T20:35:55.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 -- 172.21.15.23:0/2547146781 learned_addr learned my addr 172.21.15.23:0/2547146781 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f79292a8640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 0x7f79241029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f792a2aa640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924101fb0 0x7f7924102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 -- 172.21.15.23:0/2547146781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 msgr2=0x7f79241029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 0x7f79241029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 -- 172.21.15.23:0/2547146781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924101fb0 msgr2=0x7f7924102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924101fb0 0x7f7924102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 -- 172.21.15.23:0/2547146781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c009840 con 0x7f7924100400 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.889+0000 7f7929aa9640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.890+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79200043d0 con 0x7f7924100400 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.890+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79200052d0 con 0x7f7924100400 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.890+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7920009560 con 0x7f7924100400 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.890+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920003d00 con 0x7f7924100400 2024-03-16T20:35:55.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.890+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79241a1c80 con 0x7f7924100400 2024-03-16T20:35:55.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.891+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792410be10 con 0x7f7924100400 2024-03-16T20:35:55.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.892+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7920005440 con 0x7f7924100400 2024-03-16T20:35:55.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.892+0000 7f791affd640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 0x7f79100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.892+0000 7f79292a8640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 0x7f79100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.892+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f79200ef880 con 0x7f7924100400 2024-03-16T20:35:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.893+0000 7f79292a8640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 0x7f79100a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7924104100 tx=0x7f7914005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:55.897+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f79200b8750 con 0x7f7924100400 2024-03-16T20:35:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.079+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79241a20f0 con 0x7f7924100400 2024-03-16T20:35:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.079+0000 7f791affd640 1 -- 172.21.15.23:0/2547146781 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79200bbe10 con 0x7f7924100400 2024-03-16T20:35:56.084 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 msgr2=0x7f79100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 0x7f79100a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7924104100 tx=0x7f7914005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 msgr2=0x7f7924104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7920000c00 tx=0x7f7920005e50 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 shutdown_connections 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79100a1a30 0x7f79100a3ef0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7924101fb0 0x7f7924102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79241015f0 0x7f79241029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 --2- 172.21.15.23:0/2547146781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7924100400 0x7f7924104390 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 >> 172.21.15.23:0/2547146781 conn(0x7f79240fb990 msgr2=0x7f7924069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 shutdown_connections 2024-03-16T20:35:56.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.081+0000 7f792bd0d640 1 -- 172.21.15.23:0/2547146781 wait complete. 2024-03-16T20:35:56.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:35:56.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 -- 172.21.15.23:0/2761532742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8100930 msgr2=0x7ff4b8100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2761532742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8100930 0x7ff4b8100d50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009900 tx=0x7ff4a802bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 -- 172.21.15.23:0/2761532742 shutdown_connections 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2761532742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8102000 0x7ff4b8108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2761532742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8101640 0x7ff4b8101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2761532742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8100930 0x7ff4b8100d50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 -- 172.21.15.23:0/2761532742 >> 172.21.15.23:0/2761532742 conn(0x7ff4b80fb120 msgr2=0x7ff4b80fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 -- 172.21.15.23:0/2761532742 shutdown_connections 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.198+0000 7ff4be10c640 1 -- 172.21.15.23:0/2761532742 wait complete. 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 Processor -- start 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 -- start start 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8100930 0x7ff4b81975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8102000 0x7ff4b819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b819f0c0 con 0x7ff4b8101640 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b819f230 con 0x7ff4b8102000 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4be10c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b819f3f0 con 0x7ff4b8100930 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38666/0 (socket says 172.21.15.23:38666) 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 -- 172.21.15.23:0/2119746946 learned_addr learned my addr 172.21.15.23:0/2119746946 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b77fe640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8100930 0x7ff4b81975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b7fff640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8102000 0x7ff4b819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 -- 172.21.15.23:0/2119746946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8100930 msgr2=0x7ff4b81975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8100930 0x7ff4b81975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 -- 172.21.15.23:0/2119746946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8102000 msgr2=0x7ff4b819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8102000 0x7ff4b819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.199+0000 7ff4b6ffd640 1 -- 172.21.15.23:0/2119746946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8009560 con 0x7ff4b8101640 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4b6ffd640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00e9f0 tx=0x7ff4ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac009700 con 0x7ff4b8101640 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4ac00cd80 con 0x7ff4b8101640 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b819f6d0 con 0x7ff4b8101640 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac018920 con 0x7ff4b8101640 2024-03-16T20:35:56.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.200+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b819fbe0 con 0x7ff4b8101640 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.201+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b81077f0 con 0x7ff4b8101640 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.202+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff4ac018a80 con 0x7ff4b8101640 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.202+0000 7ff4b4ff9640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 0x7ff4a00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.202+0000 7ff4b77fe640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 0x7ff4a00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.202+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff4ac014070 con 0x7ff4b8101640 2024-03-16T20:35:56.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.203+0000 7ff4b77fe640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 0x7ff4a00a3ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8005ab0 tx=0x7ff4a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.207+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff4ac053280 con 0x7ff4b8101640 2024-03-16T20:35:56.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2889043726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:56.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:56 smithi023 ceph-mon[31879]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.388+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4b802d000 con 0x7ff4b8101640 2024-03-16T20:35:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.389+0000 7ff4b4ff9640 1 -- 172.21.15.23:0/2119746946 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4ac096a30 con 0x7ff4b8101640 2024-03-16T20:35:56.394 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 msgr2=0x7ff4a00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 0x7ff4a00a3ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8005ab0 tx=0x7ff4a8005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 msgr2=0x7ff4b8197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00e9f0 tx=0x7ff4ac00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 shutdown_connections 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff4a00a1a30 0x7ff4a00a3ef0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4b8102000 0x7ff4b819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4b8101640 0x7ff4b8197b10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 --2- 172.21.15.23:0/2119746946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4b8100930 0x7ff4b81975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.390+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 >> 172.21.15.23:0/2119746946 conn(0x7ff4b80fb120 msgr2=0x7ff4b81049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.391+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 shutdown_connections 2024-03-16T20:35:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.391+0000 7ff4be10c640 1 -- 172.21.15.23:0/2119746946 wait complete. 2024-03-16T20:35:56.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:35:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2889043726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:56 smithi031 ceph-mon[36281]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2889043726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:56 smithi189 ceph-mon[36655]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1741998181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 msgr2=0x7f39d4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1741998181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d4108830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009ca0 tx=0x7f39cc02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1741998181 shutdown_connections 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1741998181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d4108830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1741998181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41015e0 0x7f39d4101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1741998181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41003f0 0x7f39d4100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1741998181 >> 172.21.15.23:0/1741998181 conn(0x7f39d40fb980 msgr2=0x7f39d40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1741998181 shutdown_connections 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.507+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1741998181 wait complete. 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 Processor -- start 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 -- start start 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41003f0 0x7f39d4197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 0x7f39d4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d419f050 con 0x7f39d4101fa0 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d419f1c0 con 0x7f39d41015e0 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39dbb8c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d419f380 con 0x7f39d41003f0 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39da129640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39d9127640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 0x7f39d4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39da129640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38684/0 (socket says 172.21.15.23:38684) 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39da129640 1 -- 172.21.15.23:0/1134793836 learned_addr learned my addr 172.21.15.23:0/1134793836 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.508+0000 7f39d9928640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41003f0 0x7f39d4197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 -- 172.21.15.23:0/1134793836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41003f0 msgr2=0x7f39d4197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41003f0 0x7f39d4197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 -- 172.21.15.23:0/1134793836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 msgr2=0x7f39d4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 0x7f39d4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 -- 172.21.15.23:0/1134793836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc009900 con 0x7f39d4101fa0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39d9127640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 0x7f39d4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39da129640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f39cc037040 tx=0x7f39cc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc0329b0 con 0x7f39d4101fa0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d419f600 con 0x7f39d4101fa0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39cc003e50 con 0x7f39d4101fa0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc0048a0 con 0x7f39d4101fa0 2024-03-16T20:35:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.509+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d419fab0 con 0x7f39d4101fa0 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.510+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4107710 con 0x7f39d4101fa0 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.511+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f39cc00a040 con 0x7f39d4101fa0 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.512+0000 7f39caffd640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 0x7f39c00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.512+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f39cc0f0960 con 0x7f39d4101fa0 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.512+0000 7f39d9928640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 0x7f39c00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.513+0000 7f39d9928640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 0x7f39c00a3ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f39d4101440 tx=0x7f39d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.516+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f39cc0b98c0 con 0x7f39d4101fa0 2024-03-16T20:35:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.697+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39d410cf00 con 0x7f39d4101fa0 2024-03-16T20:35:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.697+0000 7f39caffd640 1 -- 172.21.15.23:0/1134793836 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39cc0bcf80 con 0x7f39d4101fa0 2024-03-16T20:35:56.702 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:35:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 msgr2=0x7f39c00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 0x7f39c00a3ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f39d4101440 tx=0x7f39d0005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 msgr2=0x7f39d419eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f39cc037040 tx=0x7f39cc009360 comp rx=0 tx=0).stop 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 shutdown_connections 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f39c00a1a30 0x7f39c00a3ef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101fa0 0x7f39d419eb10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d41015e0 0x7f39d4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 --2- 172.21.15.23:0/1134793836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39d41003f0 0x7f39d4197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.699+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 >> 172.21.15.23:0/1134793836 conn(0x7f39d40fb980 msgr2=0x7f39d41048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.700+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 shutdown_connections 2024-03-16T20:35:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.700+0000 7f39dbb8c640 1 -- 172.21.15.23:0/1134793836 wait complete. 2024-03-16T20:35:56.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 -- 172.21.15.23:0/3156912204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c1015f0 msgr2=0x7faa6c101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 --2- 172.21.15.23:0/3156912204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c1015f0 0x7faa6c101a70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faa60009900 tx=0x7faa6002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 -- 172.21.15.23:0/3156912204 shutdown_connections 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 --2- 172.21.15.23:0/3156912204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c101fb0 0x7faa6c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 --2- 172.21.15.23:0/3156912204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c1015f0 0x7faa6c101a70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 --2- 172.21.15.23:0/3156912204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 0x7faa6c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 -- 172.21.15.23:0/3156912204 >> 172.21.15.23:0/3156912204 conn(0x7faa6c0fb990 msgr2=0x7faa6c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 -- 172.21.15.23:0/3156912204 shutdown_connections 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.816+0000 7faa7420f640 1 -- 172.21.15.23:0/3156912204 wait complete. 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 Processor -- start 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 -- start start 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 0x7faa6c10acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c1015f0 0x7faa6c109300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c109fa0 con 0x7faa6c101fb0 2024-03-16T20:35:56.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c10a110 con 0x7faa6c100400 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa7420f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c1a1600 con 0x7faa6c1015f0 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa727ac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa71fab640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 0x7faa6c10acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa727ac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38696/0 (socket says 172.21.15.23:38696) 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa727ac640 1 -- 172.21.15.23:0/3197172027 learned_addr learned my addr 172.21.15.23:0/3197172027 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.817+0000 7faa717aa640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c1015f0 0x7faa6c109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 -- 172.21.15.23:0/3197172027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c1015f0 msgr2=0x7faa6c109300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c1015f0 0x7faa6c109300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 -- 172.21.15.23:0/3197172027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 msgr2=0x7faa6c10acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 0x7faa6c10acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 -- 172.21.15.23:0/3197172027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c009840 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa727ac640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009cf0 tx=0x7faa5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c009b40 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa60009560 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa5c003b70 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0108f0 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.818+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c1a1c00 con 0x7faa6c101fb0 2024-03-16T20:35:56.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.819+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c107770 con 0x7faa6c101fb0 2024-03-16T20:35:56.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.820+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faa5c003ce0 con 0x7faa6c101fb0 2024-03-16T20:35:56.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.821+0000 7faa6affd640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 0x7faa540a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:56.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.821+0000 7faa71fab640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 0x7faa540a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:56.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.821+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7faa5c014070 con 0x7faa6c101fb0 2024-03-16T20:35:56.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.821+0000 7faa71fab640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 0x7faa540a3f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa6c101450 tx=0x7faa64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:56.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:56.824+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faa5c096f90 con 0x7faa6c101fb0 2024-03-16T20:35:57.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.006+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7faa6c02d000 con 0x7faa6c101fb0 2024-03-16T20:35:57.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.007+0000 7faa6affd640 1 -- 172.21.15.23:0/3197172027 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7faa5c09a650 con 0x7faa6c101fb0 2024-03-16T20:35:57.011 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.008+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 msgr2=0x7faa540a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.008+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 0x7faa540a3f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa6c101450 tx=0x7faa64005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.008+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 msgr2=0x7faa6c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.008+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009cf0 tx=0x7faa5c00c6a0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 shutdown_connections 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7faa540a1ab0 0x7faa540a3f70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa6c101fb0 0x7faa6c109840 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa6c1015f0 0x7faa6c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 --2- 172.21.15.23:0/3197172027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c100400 0x7faa6c10acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 >> 172.21.15.23:0/3197172027 conn(0x7faa6c0fb990 msgr2=0x7faa6c104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 shutdown_connections 2024-03-16T20:35:57.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.009+0000 7faa7420f640 1 -- 172.21.15.23:0/3197172027 wait complete. 2024-03-16T20:35:57.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:35:57.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 -- 172.21.15.23:0/2424596153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 msgr2=0x7f55d0109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 --2- 172.21.15.23:0/2424596153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0109b20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009ca0 tx=0x7f55c802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 -- 172.21.15.23:0/2424596153 shutdown_connections 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 --2- 172.21.15.23:0/2424596153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0109b20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 --2- 172.21.15.23:0/2424596153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 0x7f55d00ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 --2- 172.21.15.23:0/2424596153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 0x7f55d0102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 -- 172.21.15.23:0/2424596153 >> 172.21.15.23:0/2424596153 conn(0x7f55d00fb130 msgr2=0x7f55d00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 -- 172.21.15.23:0/2424596153 shutdown_connections 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 -- 172.21.15.23:0/2424596153 wait complete. 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.123+0000 7f55d654b640 1 Processor -- start 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 -- start start 2024-03-16T20:35:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 0x7f55d0197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 0x7f55d019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d019f040 con 0x7f55d00fff70 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d019f1b0 con 0x7f55d00ff5b0 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d654b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d019f370 con 0x7f55d0102b40 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38702/0 (socket says 172.21.15.23:38702) 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 -- 172.21.15.23:0/1818427197 learned_addr learned my addr 172.21.15.23:0/1818427197 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cffff640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 0x7f55d0197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55d4ae8640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 0x7f55d019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 -- 172.21.15.23:0/1818427197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 msgr2=0x7f55d019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 0x7f55d019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 -- 172.21.15.23:0/1818427197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 msgr2=0x7f55d0197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 0x7f55d0197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 -- 172.21.15.23:0/1818427197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4009490 con 0x7f55d00fff70 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cf7fe640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f55c400ea60 tx=0x7f55c400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.124+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400cde0 con 0x7f55d00fff70 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.125+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55c40035f0 con 0x7f55d00fff70 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.125+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c8009900 con 0x7f55d00fff70 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.125+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4010690 con 0x7f55d00fff70 2024-03-16T20:35:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.125+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d019f9d0 con 0x7f55d00fff70 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.126+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d01089a0 con 0x7f55d00fff70 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.127+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f55c4003af0 con 0x7f55d00fff70 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.127+0000 7f55cd7fa640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 0x7f55b80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.127+0000 7f55cffff640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 0x7f55b80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.127+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f55c4014070 con 0x7f55d00fff70 2024-03-16T20:35:57.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.128+0000 7f55cffff640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 0x7f55b80a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55d0069860 tx=0x7f55c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.132+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f55c4096a80 con 0x7f55d00fff70 2024-03-16T20:35:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2547146781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2119746946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1134793836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3197172027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.352+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55d010cfa0 con 0x7f55d00fff70 2024-03-16T20:35:57.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.353+0000 7f55cd7fa640 1 -- 172.21.15.23:0/1818427197 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7f55c409a140 con 0x7f55d00fff70 2024-03-16T20:35:57.358 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:57.358 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:35:57.358 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 msgr2=0x7f55b80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 0x7f55b80a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55d0069860 tx=0x7f55c0005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 msgr2=0x7f55d0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f55c400ea60 tx=0x7f55c400ef20 comp rx=0 tx=0).stop 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 shutdown_connections 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f55b80a1a30 0x7f55b80a3ef0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55d0102b40 0x7f55d019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d00fff70 0x7f55d0197a90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 --2- 172.21.15.23:0/1818427197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d00ff5b0 0x7f55d0197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 >> 172.21.15.23:0/1818427197 conn(0x7f55d00fb130 msgr2=0x7f55d0105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 shutdown_connections 2024-03-16T20:35:57.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.355+0000 7f55d654b640 1 -- 172.21.15.23:0/1818427197 wait complete. 2024-03-16T20:35:57.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:57.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2547146781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2119746946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1134793836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3197172027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2547146781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2119746946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1134793836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3197172027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 -- 172.21.15.23:0/3189143691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec107270 msgr2=0x7f6bec109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3189143691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec107270 0x7f6bec109660 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009be0 tx=0x7f6be802bef0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 -- 172.21.15.23:0/3189143691 shutdown_connections 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3189143691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec107270 0x7f6bec109660 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3189143691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec104940 0x7f6bec106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3189143691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec104400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.478+0000 7f6bf3284640 1 -- 172.21.15.23:0/3189143691 >> 172.21.15.23:0/3189143691 conn(0x7f6bec0fb980 msgr2=0x7f6bec0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- 172.21.15.23:0/3189143691 shutdown_connections 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- 172.21.15.23:0/3189143691 wait complete. 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 Processor -- start 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- start start 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 0x7f6bec197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec19f090 con 0x7f6bec107270 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec19f200 con 0x7f6bec104940 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf3284640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec19f3c0 con 0x7f6bec102010 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.479+0000 7f6bf1821640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38718/0 (socket says 172.21.15.23:38718) 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 -- 172.21.15.23:0/3978544060 learned_addr learned my addr 172.21.15.23:0/3978544060 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1020640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf081f640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 0x7f6bec197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 -- 172.21.15.23:0/3978544060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 msgr2=0x7f6bec1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 -- 172.21.15.23:0/3978544060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 msgr2=0x7f6bec197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 0x7f6bec197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 -- 172.21.15.23:0/3978544060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc009560 con 0x7f6bec107270 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1020640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf081f640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 0x7f6bec197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf1821640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6be8004af0 tx=0x7f6be8033d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8003d80 con 0x7f6bec107270 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6be802ce40 con 0x7f6bec107270 2024-03-16T20:35:57.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8009840 con 0x7f6bec107270 2024-03-16T20:35:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8034960 con 0x7f6bec107270 2024-03-16T20:35:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.480+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec19f960 con 0x7f6bec107270 2024-03-16T20:35:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.481+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec0fd470 con 0x7f6bec107270 2024-03-16T20:35:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.483+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6be8034ac0 con 0x7f6bec107270 2024-03-16T20:35:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.483+0000 7f6be27fc640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 0x7f6bd80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.484+0000 7f6bf1020640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 0x7f6bd80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.484+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f6be80efe20 con 0x7f6bec107270 2024-03-16T20:35:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.484+0000 7f6bf1020640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 0x7f6bd80a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009fc0 tx=0x7f6be4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.487+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6be80b8da0 con 0x7f6bec107270 2024-03-16T20:35:57.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.670+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bec02d000 con 0x7f6bec107270 2024-03-16T20:35:57.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.672+0000 7f6be27fc640 1 -- 172.21.15.23:0/3978544060 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f6be80bc460 con 0x7f6bec107270 2024-03-16T20:35:57.677 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:57.677 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:57.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 msgr2=0x7f6bd80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 0x7f6bd80a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009fc0 tx=0x7f6be4005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 msgr2=0x7f6bec19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6be8004af0 tx=0x7f6be8033d00 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 shutdown_connections 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f6bd80a1a30 0x7f6bd80a3ef0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bec107270 0x7f6bec19eb50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bec104940 0x7f6bec197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 --2- 172.21.15.23:0/3978544060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bec102010 0x7f6bec1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 >> 172.21.15.23:0/3978544060 conn(0x7f6bec0fb980 msgr2=0x7f6bec1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 shutdown_connections 2024-03-16T20:35:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.674+0000 7f6bf3284640 1 -- 172.21.15.23:0/3978544060 wait complete. 2024-03-16T20:35:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:35:57.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 -- 172.21.15.23:0/1347643039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff550101510 msgr2=0x7ff550101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 --2- 172.21.15.23:0/1347643039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff550101510 0x7ff550101930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff544009900 tx=0x7ff54402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 -- 172.21.15.23:0/1347643039 shutdown_connections 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 --2- 172.21.15.23:0/1347643039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5501030c0 0x7ff550109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 --2- 172.21.15.23:0/1347643039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 0x7ff550102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 --2- 172.21.15.23:0/1347643039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff550101510 0x7ff550101930 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 -- 172.21.15.23:0/1347643039 >> 172.21.15.23:0/1347643039 conn(0x7ff5500fca80 msgr2=0x7ff5500feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 -- 172.21.15.23:0/1347643039 shutdown_connections 2024-03-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.791+0000 7ff556e55640 1 -- 172.21.15.23:0/1347643039 wait complete. 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 Processor -- start 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 -- start start 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550101510 0x7ff5501975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 0x7ff550197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019f0a0 con 0x7ff5501030c0 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019f210 con 0x7ff550102700 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55019f3d0 con 0x7ff550101510 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38748/0 (socket says 172.21.15.23:38748) 2024-03-16T20:35:57.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 -- 172.21.15.23:0/234106261 learned_addr learned my addr 172.21.15.23:0/234106261 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff554bf1640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550101510 0x7ff5501975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff54ffff640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 0x7ff550197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 -- 172.21.15.23:0/234106261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550101510 msgr2=0x7ff5501975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550101510 0x7ff5501975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 -- 172.21.15.23:0/234106261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 msgr2=0x7ff550197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 0x7ff550197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 -- 172.21.15.23:0/234106261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff548009840 con 0x7ff5501030c0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff5553f2640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff54800efd0 tx=0x7ff54800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548009b40 con 0x7ff5501030c0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff548003b70 con 0x7ff5501030c0 2024-03-16T20:35:57.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.792+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff544009560 con 0x7ff5501030c0 2024-03-16T20:35:57.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.793+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5480108f0 con 0x7ff5501030c0 2024-03-16T20:35:57.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.793+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55019fa30 con 0x7ff5501030c0 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.794+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff550108850 con 0x7ff5501030c0 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.795+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff548003ce0 con 0x7ff5501030c0 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.795+0000 7ff54dffb640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 0x7ff5400a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.795+0000 7ff554bf1640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 0x7ff5400a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.795+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff548014070 con 0x7ff5501030c0 2024-03-16T20:35:57.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.796+0000 7ff554bf1640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 0x7ff5400a3f70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff54402c240 tx=0x7ff54402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:57.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.799+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff548096b50 con 0x7ff5501030c0 2024-03-16T20:35:57.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.981+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff55002d000 con 0x7ff5501030c0 2024-03-16T20:35:57.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.981+0000 7ff54dffb640 1 -- 172.21.15.23:0/234106261 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff54809a210 con 0x7ff5501030c0 2024-03-16T20:35:57.986 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:35:57.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 msgr2=0x7ff5400a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 0x7ff5400a3f70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff54402c240 tx=0x7ff54402c7e0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 msgr2=0x7ff55019eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff54800efd0 tx=0x7ff54800c6a0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 shutdown_connections 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff5400a1ab0 0x7ff5400a3f70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5501030c0 0x7ff55019eb60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff550102700 0x7ff550197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 --2- 172.21.15.23:0/234106261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff550101510 0x7ff5501975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 >> 172.21.15.23:0/234106261 conn(0x7ff5500fca80 msgr2=0x7ff550105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 shutdown_connections 2024-03-16T20:35:57.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:57.984+0000 7ff556e55640 1 -- 172.21.15.23:0/234106261 wait complete. 2024-03-16T20:35:57.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:35:58.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/1457987378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101fc0 msgr2=0x7fd9cc108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/1457987378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc108850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009ca0 tx=0x7fd9c402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/1457987378 shutdown_connections 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/1457987378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc108850 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/1457987378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc101600 0x7fd9cc101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/1457987378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc100410 0x7fd9cc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/1457987378 >> 172.21.15.23:0/1457987378 conn(0x7fd9cc0fb9a0 msgr2=0x7fd9cc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/1457987378 shutdown_connections 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/1457987378 wait complete. 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 Processor -- start 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- start start 2024-03-16T20:35:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc100410 0x7fd9cc06a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19f040 con 0x7fd9cc101600 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19f1b0 con 0x7fd9cc100410 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.102+0000 7fd9d2b2c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19f370 con 0x7fd9cc101fc0 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38770/0 (socket says 172.21.15.23:38770) 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 -- 172.21.15.23:0/2096695172 learned_addr learned my addr 172.21.15.23:0/2096695172 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9d10c9640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9d08c8640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc100410 0x7fd9cc06a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 -- 172.21.15.23:0/2096695172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc101fc0 msgr2=0x7fd9cc19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 -- 172.21.15.23:0/2096695172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc100410 msgr2=0x7fd9cc06a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc100410 0x7fd9cc06a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 -- 172.21.15.23:0/2096695172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b8009490 con 0x7fd9cc101600 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9cbfff640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ea60 tx=0x7fd9b800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b800cde0 con 0x7fd9cc101600 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9b80035f0 con 0x7fd9cc101600 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c4009900 con 0x7fd9cc101600 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.103+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8010690 con 0x7fd9cc101600 2024-03-16T20:35:58.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.104+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc19fa20 con 0x7fd9cc101600 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.105+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc1077b0 con 0x7fd9cc101600 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.106+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd9b8003af0 con 0x7fd9cc101600 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.106+0000 7fd9c9ffb640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 0x7fd9bc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.106+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fd9b8014070 con 0x7fd9cc101600 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.106+0000 7fd9d08c8640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 0x7fd9bc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.107+0000 7fd9d08c8640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 0x7fd9bc0a3f70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0005fd0 tx=0x7fd9c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.111+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd9b8096970 con 0x7fd9cc101600 2024-03-16T20:35:58.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.291+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9cc02d000 con 0x7fd9cc101600 2024-03-16T20:35:58.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.292+0000 7fd9c9ffb640 1 -- 172.21.15.23:0/2096695172 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9b809a030 con 0x7fd9cc101600 2024-03-16T20:35:58.296 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 msgr2=0x7fd9bc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 0x7fd9bc0a3f70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0005fd0 tx=0x7fd9c000b410 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 msgr2=0x7fd9cc19c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ea60 tx=0x7fd9b800ef20 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 shutdown_connections 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9bc0a1ab0 0x7fd9bc0a3f70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9cc101fc0 0x7fd9cc19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9cc101600 0x7fd9cc19c270 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 --2- 172.21.15.23:0/2096695172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9cc100410 0x7fd9cc06a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 >> 172.21.15.23:0/2096695172 conn(0x7fd9cc0fb9a0 msgr2=0x7fd9cc104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 shutdown_connections 2024-03-16T20:35:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.294+0000 7fd9d2b2c640 1 -- 172.21.15.23:0/2096695172 wait complete. 2024-03-16T20:35:58.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1818427197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3978544060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:58 smithi023 ceph-mon[31879]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:58.308 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/234106261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:58.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:35:58.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 -- 172.21.15.23:0/2209480680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101fc0 msgr2=0x7f0b94108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 --2- 172.21.15.23:0/2209480680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101fc0 0x7f0b94108850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009ca0 tx=0x7f0b8c02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 -- 172.21.15.23:0/2209480680 shutdown_connections 2024-03-16T20:35:58.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 --2- 172.21.15.23:0/2209480680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101fc0 0x7f0b94108850 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 --2- 172.21.15.23:0/2209480680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101600 0x7f0b94101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 --2- 172.21.15.23:0/2209480680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b94100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 -- 172.21.15.23:0/2209480680 >> 172.21.15.23:0/2209480680 conn(0x7f0b940fb9a0 msgr2=0x7f0b940fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 -- 172.21.15.23:0/2209480680 shutdown_connections 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.410+0000 7f0b9b688640 1 -- 172.21.15.23:0/2209480680 wait complete. 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 Processor -- start 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 -- start start 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b9406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101fc0 0x7f0b9419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9419f040 con 0x7f0b94101600 2024-03-16T20:35:58.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9419f1b0 con 0x7f0b94100410 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b9b688640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9419f370 con 0x7f0b94101fc0 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b98c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b9406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b98c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38792/0 (socket says 172.21.15.23:38792) 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b98c23640 1 -- 172.21.15.23:0/3870963113 learned_addr learned my addr 172.21.15.23:0/3870963113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99c25640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101fc0 0x7f0b9419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 -- 172.21.15.23:0/3870963113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101fc0 msgr2=0x7f0b9419e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101fc0 0x7f0b9419e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 -- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 msgr2=0x7f0b9419c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b99424640 1 -- 172.21.15.23:0/3870963113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c009900 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.411+0000 7f0b98c23640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b99424640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b9406a6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000e9c0 tx=0x7f0b9000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9000ccc0 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9419f650 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b9000ce20 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90010430 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.412+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9419fbb0 con 0x7f0b94100410 2024-03-16T20:35:58.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.413+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b941076d0 con 0x7f0b94100410 2024-03-16T20:35:58.496 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1818427197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.496 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3978544060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.496 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:58 smithi189 ceph-mon[36655]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:58.497 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/234106261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:58.497 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1818427197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.497 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3978544060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:35:58.497 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:58 smithi031 ceph-mon[36281]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:35:58.497 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/234106261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:35:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.414+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0b90003f40 con 0x7f0b94100410 2024-03-16T20:35:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.415+0000 7f0b8a7fc640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 0x7f0b800a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.415+0000 7f0b98c23640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 0x7f0b800a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.415+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f0b90014070 con 0x7f0b94100410 2024-03-16T20:35:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.415+0000 7f0b98c23640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 0x7f0b800a3ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0b9406a440 tx=0x7f0b84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.418+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0b90096c20 con 0x7f0b94100410 2024-03-16T20:35:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.598+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b9410be20 con 0x7f0b94100410 2024-03-16T20:35:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.598+0000 7f0b8a7fc640 1 -- 172.21.15.23:0/3870963113 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b9009a2e0 con 0x7f0b94100410 2024-03-16T20:35:58.603 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:35:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 msgr2=0x7f0b800a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 0x7f0b800a3ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0b9406a440 tx=0x7f0b84005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 msgr2=0x7f0b9406a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b9406a6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000e9c0 tx=0x7f0b9000ee80 comp rx=0 tx=0).stop 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 shutdown_connections 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f0b800a1a30 0x7f0b800a3ef0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b94101fc0 0x7f0b9419e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b94101600 0x7f0b9419c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 --2- 172.21.15.23:0/3870963113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b94100410 0x7f0b9406a6d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.600+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 >> 172.21.15.23:0/3870963113 conn(0x7f0b940fb9a0 msgr2=0x7f0b941048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.601+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 shutdown_connections 2024-03-16T20:35:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.601+0000 7f0b9b688640 1 -- 172.21.15.23:0/3870963113 wait complete. 2024-03-16T20:35:58.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 -- 172.21.15.23:0/1936681716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc100400 msgr2=0x7f05bc100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 --2- 172.21.15.23:0/1936681716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc100400 0x7f05bc100820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05ac009900 tx=0x7f05ac02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 -- 172.21.15.23:0/1936681716 shutdown_connections 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 --2- 172.21.15.23:0/1936681716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc101fb0 0x7f05bc108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 --2- 172.21.15.23:0/1936681716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc1015f0 0x7f05bc101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 --2- 172.21.15.23:0/1936681716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc100400 0x7f05bc100820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 -- 172.21.15.23:0/1936681716 >> 172.21.15.23:0/1936681716 conn(0x7f05bc0fb990 msgr2=0x7f05bc0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 -- 172.21.15.23:0/1936681716 shutdown_connections 2024-03-16T20:35:58.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.717+0000 7f05c0927640 1 -- 172.21.15.23:0/1936681716 wait complete. 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 Processor -- start 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 -- start start 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc1015f0 0x7f05bc1029d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc101fb0 0x7f05bc102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc103670 con 0x7f05bc100400 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1037e0 con 0x7f05bc1015f0 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05c0927640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc1a1610 con 0x7f05bc101fb0 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38816/0 (socket says 172.21.15.23:38816) 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 -- 172.21.15.23:0/2970777054 learned_addr learned my addr 172.21.15.23:0/2970777054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05b9d9b640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc1015f0 0x7f05bc1029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05bad9d640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc101fb0 0x7f05bc102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 -- 172.21.15.23:0/2970777054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc101fb0 msgr2=0x7f05bc102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc101fb0 0x7f05bc102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 -- 172.21.15.23:0/2970777054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc1015f0 msgr2=0x7f05bc1029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc1015f0 0x7f05bc1029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.718+0000 7f05ba59c640 1 -- 172.21.15.23:0/2970777054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac009560 con 0x7f05bc100400 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05ba59c640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000c00 tx=0x7f05ac005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac0043d0 con 0x7f05bc100400 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05ac0052d0 con 0x7f05bc100400 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc1a1860 con 0x7f05bc100400 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac003d70 con 0x7f05bc100400 2024-03-16T20:35:58.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.719+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc1a1d70 con 0x7f05bc100400 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.720+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc10be10 con 0x7f05bc100400 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.721+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f05ac005440 con 0x7f05bc100400 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.722+0000 7f05ab7fe640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 0x7f05a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.722+0000 7f05b9d9b640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 0x7f05a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.722+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f05ac0ef840 con 0x7f05bc100400 2024-03-16T20:35:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.722+0000 7f05b9d9b640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 0x7f05a40a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f05bc104100 tx=0x7f05b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.726+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f05ac0b8710 con 0x7f05bc100400 2024-03-16T20:35:58.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.907+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f05bc10bfa0 con 0x7f05bc100400 2024-03-16T20:35:58.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.908+0000 7f05ab7fe640 1 -- 172.21.15.23:0/2970777054 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f05bc10bfa0 con 0x7f05bc100400 2024-03-16T20:35:58.912 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:35:58.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 msgr2=0x7f05a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 0x7f05a40a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f05bc104100 tx=0x7f05b0005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 msgr2=0x7f05bc104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000c00 tx=0x7f05ac005e50 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 shutdown_connections 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f05a40a1a30 0x7f05a40a3ef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05bc101fb0 0x7f05bc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05bc1015f0 0x7f05bc1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 --2- 172.21.15.23:0/2970777054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc100400 0x7f05bc104390 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 >> 172.21.15.23:0/2970777054 conn(0x7f05bc0fb990 msgr2=0x7f05bc0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 shutdown_connections 2024-03-16T20:35:58.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:58.910+0000 7f05c0927640 1 -- 172.21.15.23:0/2970777054 wait complete. 2024-03-16T20:35:58.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:35:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 -- 172.21.15.23:0/2154380506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100550 msgr2=0x7f32fc1009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 --2- 172.21.15.23:0/2154380506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100550 0x7f32fc1009d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009900 tx=0x7f32e402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 -- 172.21.15.23:0/2154380506 shutdown_connections 2024-03-16T20:35:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 --2- 172.21.15.23:0/2154380506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100f10 0x7f32fc101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 --2- 172.21.15.23:0/2154380506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100550 0x7f32fc1009d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 --2- 172.21.15.23:0/2154380506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc0ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 -- 172.21.15.23:0/2154380506 >> 172.21.15.23:0/2154380506 conn(0x7f32fc0fb120 msgr2=0x7f32fc0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 -- 172.21.15.23:0/2154380506 shutdown_connections 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.023+0000 7f3300c5c640 1 -- 172.21.15.23:0/2154380506 wait complete. 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 Processor -- start 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 -- start start 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100550 0x7f32fc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc1a12d0 con 0x7f32fc100f10 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc1a1440 con 0x7f32fc100550 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f3300c5c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc1a1600 con 0x7f32fc0ff7f0 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32fad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32fa59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32fad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38834/0 (socket says 172.21.15.23:38834) 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32fad9d640 1 -- 172.21.15.23:0/214775755 learned_addr learned my addr 172.21.15.23:0/214775755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32f3fff640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100550 0x7f32fc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.024+0000 7f32fa59c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc1997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:50816/0 (socket says 172.21.15.23:50816) 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 -- 172.21.15.23:0/214775755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 msgr2=0x7f32fc1997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc1997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 -- 172.21.15.23:0/214775755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100550 msgr2=0x7f32fc199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100550 0x7f32fc199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 -- 172.21.15.23:0/214775755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4009560 con 0x7f32fc100f10 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32fad9d640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f32f400eb10 tx=0x7f32f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4004b20 con 0x7f32fc100f10 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc1a18e0 con 0x7f32fc100f10 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32f4004c80 con 0x7f32fc100f10 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4010650 con 0x7f32fc100f10 2024-03-16T20:35:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.025+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc1a1dc0 con 0x7f32fc100f10 2024-03-16T20:35:59.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.026+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc10b4e0 con 0x7f32fc100f10 2024-03-16T20:35:59.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.027+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f32f4005020 con 0x7f32fc100f10 2024-03-16T20:35:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.028+0000 7f32f37fe640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 0x7f32e80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.028+0000 7f32fa59c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 0x7f32e80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.028+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f32f4014070 con 0x7f32fc100f10 2024-03-16T20:35:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.028+0000 7f32fa59c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 0x7f32e80a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009bd0 tx=0x7f32ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.032+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f32f4096bd0 con 0x7f32fc100f10 2024-03-16T20:35:59.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.253+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32fc02d000 con 0x7f32fc100f10 2024-03-16T20:35:59.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.254+0000 7f32f37fe640 1 -- 172.21.15.23:0/214775755 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11449 (secure 0 0 0) 0x7f32f409a290 con 0x7f32fc100f10 2024-03-16T20:35:59.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:59.259 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:34:10.946679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24497,24511,24418,24493,24416]},"id":1}]} 2024-03-16T20:35:59.259 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 msgr2=0x7f32e80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 0x7f32e80a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009bd0 tx=0x7f32ec005f90 comp rx=0 tx=0).stop 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 msgr2=0x7f32fc1a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f32f400eb10 tx=0x7f32f400c2d0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 shutdown_connections 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32e80a1a30 0x7f32e80a3ef0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32fc100f10 0x7f32fc1a0d90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32fc100550 0x7f32fc199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 --2- 172.21.15.23:0/214775755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32fc0ff7f0 0x7f32fc1997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 >> 172.21.15.23:0/214775755 conn(0x7f32fc0fb120 msgr2=0x7f32fc0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 shutdown_connections 2024-03-16T20:35:59.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.256+0000 7f3300c5c640 1 -- 172.21.15.23:0/214775755 wait complete. 2024-03-16T20:35:59.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2096695172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3870963113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:35:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2970777054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:35:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- 172.21.15.23:0/3941672171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac107270 msgr2=0x7f4bac1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/3941672171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac107270 0x7f4bac1096c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009900 tx=0x7f4b9c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- 172.21.15.23:0/3941672171 shutdown_connections 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/3941672171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac107270 0x7f4bac1096c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/3941672171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac104940 0x7f4bac106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/3941672171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 0x7f4bac104400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- 172.21.15.23:0/3941672171 >> 172.21.15.23:0/3941672171 conn(0x7f4bac0fba20 msgr2=0x7f4bac0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- 172.21.15.23:0/3941672171 shutdown_connections 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- 172.21.15.23:0/3941672171 wait complete. 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 Processor -- start 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.371+0000 7f4bb2df7640 1 -- start start 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 0x7f4bac197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac107270 0x7f4bac19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac19f050 con 0x7f4bac104940 2024-03-16T20:35:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac19f1c0 con 0x7f4bac102010 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac19f380 con 0x7f4bac107270 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38840/0 (socket says 172.21.15.23:38840) 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 -- 172.21.15.23:0/1049765470 learned_addr learned my addr 172.21.15.23:0/1049765470 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb1394640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac107270 0x7f4bac19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb0b93640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 0x7f4bac197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 -- 172.21.15.23:0/1049765470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac107270 msgr2=0x7f4bac19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac107270 0x7f4bac19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 -- 172.21.15.23:0/1049765470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 msgr2=0x7f4bac197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 0x7f4bac197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 -- 172.21.15.23:0/1049765470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4009840 con 0x7f4bac104940 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4babfff640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00096c0 con 0x7f4bac104940 2024-03-16T20:35:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0005500 con 0x7f4bac104940 2024-03-16T20:35:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0010430 con 0x7f4bac104940 2024-03-16T20:35:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.372+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c009560 con 0x7f4bac104940 2024-03-16T20:35:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.373+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac19f9e0 con 0x7f4bac104940 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.373+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac0fd510 con 0x7f4bac104940 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.374+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4ba0005020 con 0x7f4bac104940 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.375+0000 7f4ba9ffb640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 0x7f4b940a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.375+0000 7f4bb0b93640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 0x7f4b940a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.375+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4ba0014070 con 0x7f4bac104940 2024-03-16T20:35:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.376+0000 7f4bb0b93640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 0x7f4b940a3f70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005fd0 tx=0x7f4ba4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.380+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4ba00968f0 con 0x7f4bac104940 2024-03-16T20:35:59.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2096695172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3870963113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:35:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2970777054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2096695172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3870963113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:35:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2970777054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:35:59.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.563+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bac0fcf80 con 0x7f4bac104940 2024-03-16T20:35:59.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.565+0000 7f4ba9ffb640 1 -- 172.21.15.23:0/1049765470 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f4ba0099fb0 con 0x7f4bac104940 2024-03-16T20:35:59.569 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:35:59.570 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 msgr2=0x7f4b940a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 0x7f4b940a3f70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005fd0 tx=0x7f4ba4009500 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 msgr2=0x7f4bac197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 shutdown_connections 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f4b940a1ab0 0x7f4b940a3f70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bac107270 0x7f4bac19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac104940 0x7f4bac197aa0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 --2- 172.21.15.23:0/1049765470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4bac102010 0x7f4bac197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 >> 172.21.15.23:0/1049765470 conn(0x7f4bac0fba20 msgr2=0x7f4bac102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 shutdown_connections 2024-03-16T20:35:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.567+0000 7f4bb2df7640 1 -- 172.21.15.23:0/1049765470 wait complete. 2024-03-16T20:35:59.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-16T20:33:49.239658+0000', 'modified': '2024-03-16T20:34:10.946679+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24511, 'mds_1': 24497, 'mds_2': 24493, 'mds_3': 24418, 'mds_4': 24416}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14643': {'gid': 14643, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/2750734224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 2750734224}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 2750734224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14655': {'gid': 14655, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6841/4113023748', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 4113023748}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 4113023748}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24416': {'gid': 24416, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.31:6833/1919282692', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6832', 'nonce': 1919282692}, {'type': 'v1', 'addr': '172.21.15.31:6833', 'nonce': 1919282692}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6833/1083314774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 1083314774}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 1083314774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6835/354681253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6834', 'nonce': 354681253}, {'type': 'v1', 'addr': '172.21.15.31:6835', 'nonce': 354681253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24469': {'gid': 24469, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1658699703', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1658699703}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1658699703}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/3204120502', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3204120502}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3204120502}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24493': {'gid': 24493, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.31:6837/424329834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6836', 'nonce': 424329834}, {'type': 'v1', 'addr': '172.21.15.31:6837', 'nonce': 424329834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.31:6839/37663771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 37663771}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 37663771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.189:6839/2333060283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2333060283}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2333060283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24511, 'qdb_cluster': [24497, 24511, 24418, 24493, 24416]} 2024-03-16T20:35:59.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-16T20:35:59.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 -- 172.21.15.23:0/3803256423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541026f0 msgr2=0x7f3254102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 --2- 172.21.15.23:0/3803256423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541026f0 0x7f3254102b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3244009900 tx=0x7f324402bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 -- 172.21.15.23:0/3803256423 shutdown_connections 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 --2- 172.21.15.23:0/3803256423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32541030b0 0x7f3254109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 --2- 172.21.15.23:0/3803256423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541026f0 0x7f3254102b70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 --2- 172.21.15.23:0/3803256423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3254101500 0x7f3254101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 -- 172.21.15.23:0/3803256423 >> 172.21.15.23:0/3803256423 conn(0x7f32540fca90 msgr2=0x7f32540feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 -- 172.21.15.23:0/3803256423 shutdown_connections 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.683+0000 7f325bb5e640 1 -- 172.21.15.23:0/3803256423 wait complete. 2024-03-16T20:35:59.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 Processor -- start 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 -- start start 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3254101500 0x7f32541975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 0x7f3254197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325419f090 con 0x7f32541030b0 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325419f200 con 0x7f32541026f0 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325bb5e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325419f3c0 con 0x7f3254101500 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38850/0 (socket says 172.21.15.23:38850) 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 -- 172.21.15.23:0/1196502196 learned_addr learned my addr 172.21.15.23:0/1196502196 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f32590f9640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 0x7f3254197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f32598fa640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3254101500 0x7f32541975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 -- 172.21.15.23:0/1196502196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3254101500 msgr2=0x7f32541975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3254101500 0x7f32541975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 -- 172.21.15.23:0/1196502196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 msgr2=0x7f3254197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 0x7f3254197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f325a0fb640 1 -- 172.21.15.23:0/1196502196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250009490 con 0x7f32541030b0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.684+0000 7f32590f9640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 0x7f3254197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f325a0fb640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f324c00eb10 tx=0x7f324c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f324affd640 1 -- 172.21.15.23:0/1196502196 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c004b20 con 0x7f32541030b0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f324affd640 1 -- 172.21.15.23:0/1196502196 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f324c004c80 con 0x7f32541030b0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3244009560 con 0x7f32541030b0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f324affd640 1 -- 172.21.15.23:0/1196502196 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c010650 con 0x7f32541030b0 2024-03-16T20:35:59.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.685+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325419f9f0 con 0x7f32541030b0 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 Processor -- start 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f324affd640 1 -- 172.21.15.23:0/1196502196 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f324c005020 con 0x7f32541030b0 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 -- start start 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 0x5635bdf3b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf3bef0 0x5635bdf3e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdeedcd0 con 0x5635bdf36390 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdea0f20 con 0x5635bdf3bef0 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f325e81a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bde9e450 con 0x5635bdf36d80 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f32598fa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 0x5635bdf3b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf367b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38862/0 (socket says 172.21.15.23:38862) 2024-03-16T20:35:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.688+0000 7f32590f9640 1 -- 172.21.15.23:0/1671999639 learned_addr learned my addr 172.21.15.23:0/1671999639 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f324affd640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 0x7f32400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32590f9640 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf3bef0 0x5635bdf3e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f325a0fb640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 0x7f32400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 -- 172.21.15.23:0/1671999639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf3bef0 msgr2=0x5635bdf3e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf3bef0 0x5635bdf3e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 -- 172.21.15.23:0/1671999639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 msgr2=0x5635bdf367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 -- 172.21.15.23:0/1671999639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635bde4ef20 con 0x5635bdf36d80 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f324affd640 1 -- 172.21.15.23:0/1196502196 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f324c014070 con 0x7f32541030b0 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f32598fa640 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 0x5635bdf3b880 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d50 tx=0x7f3250011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5fc693ca4d31870 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f325a0fb640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 0x7f32400a3ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f324c096d20 tx=0x7f324c069cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f323ffff640 1 -- 172.21.15.23:0/1671999639 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250004230 con 0x5635bdf36d80 2024-03-16T20:35:59.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f323ffff640 1 -- 172.21.15.23:0/1671999639 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3250004390 con 0x5635bdf36d80 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.689+0000 7f323ffff640 1 -- 172.21.15.23:0/1671999639 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250004e30 con 0x5635bdf36d80 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- 172.21.15.23:0/1671999639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 msgr2=0x5635bdf3b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 0x5635bdf3b880 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d50 tx=0x7f3250011810 comp rx=0 tx=0).stop 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- 172.21.15.23:0/1671999639 shutdown_connections 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf3bef0 0x5635bdf3e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36d80 0x5635bdf3b880 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 --2- 172.21.15.23:0/1671999639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- 172.21.15.23:0/1671999639 >> 172.21.15.23:0/1671999639 conn(0x5635bdf2d4f0 msgr2=0x5635bdf2d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- 172.21.15.23:0/1671999639 shutdown_connections 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- 172.21.15.23:0/1671999639 wait complete. 2024-03-16T20:35:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 -- start start 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.690+0000 7f325e81a740 1 Processor -- start 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36390 0x5635bdf404b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 0x5635bdf47a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bde99c30 con 0x5635bdf3bef0 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bde3ed30 con 0x5635bdf36d80 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdb6d310 con 0x5635bdf36390 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f325e81a740 10 client.0 _ll_register_callbacks cb 0x5635bde46d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 0x5635bdf47a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 0x5635bdf47a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38868/0 (socket says 172.21.15.23:38868) 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32590f9640 1 -- 172.21.15.23:0/913461387 learned_addr learned my addr 172.21.15.23:0/913461387 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32590f9640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36390 0x5635bdf404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 -- 172.21.15.23:0/913461387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36390 msgr2=0x5635bdf404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36390 0x5635bdf404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 -- 172.21.15.23:0/913461387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 msgr2=0x5635bdf47a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 0x5635bdf47a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 -- 172.21.15.23:0/913461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32500099b0 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f32598fa640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf409f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5635bdf2e460 tx=0x7f3250005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ae41f22e1f371a server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.691+0000 7f323d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250019800 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635bdb996c0 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mon.2 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3250019960 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3250004500 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 10 client.25223 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5635bde982b0 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mon.2 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5635bdb996c0 con 0x5635bdf36d80 2024-03-16T20:35:59.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mon.2 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f32500249b0 con 0x5635bdf36d80 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 10 client.25223 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 10 client.25223 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 20 client.25223 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdf4cd70 0x5635bdf4c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 4 client.25223 mds_command: new command op to 24416 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 --> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635bdbea270 con 0x5635bdf4cd70 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.692+0000 7f32590f9640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdf4cd70 0x5635bdf4c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.693+0000 7f323d7fa640 10 client.25223 ms_handle_connect on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.693+0000 7f32590f9640 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdf4cd70 0x5635bdf4c020 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5f893446d36ada1e server_cookie=17804e3bf31fab1a in_seq=0 out_seq=0 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f323d7fa640 1 -- 172.21.15.23:0/913461387 <== mds.4 v2:172.21.15.31:6832/1919282692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635bdbea270 con 0x5635bdf4cd70 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f323d7fa640 10 client.25223 handle_command_reply: tid=0 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 client.25223 shutdown 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 msgr2=0x5635bdf409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf409f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5635bdf2e460 tx=0x7f3250005340 comp rx=0 tx=0).stop 2024-03-16T20:35:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 shutdown_connections 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf3bef0 0x5635bdf47a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdf4cd70 0x5635bdf4c020 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf409f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 --2- 172.21.15.23:0/913461387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf36390 0x5635bdf404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f323d7fa640 0 client.25223 ms_handle_reset on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 >> 172.21.15.23:0/913461387 conn(0x5635bdf2d4f0 msgr2=0x5635bdf2c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 shutdown_connections 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 1 -- 172.21.15.23:0/913461387 wait complete. 2024-03-16T20:35:59.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.694+0000 7f325e81a740 20 client.25223 trim_cache size 0 max 16384 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 Processor -- start 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 -- start start 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf3f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bde1f490 con 0x5635bdf36390 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdf3ea40 con 0x5635bdf36d80 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f325e81a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdbea270 con 0x5635bdf3bef0 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38880/0 (socket says 172.21.15.23:38880) 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 -- 172.21.15.23:0/2272858954 learned_addr learned my addr 172.21.15.23:0/2272858954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32598fa640 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf3f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 -- 172.21.15.23:0/2272858954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 msgr2=0x5635bdf3f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf3f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 -- 172.21.15.23:0/2272858954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 msgr2=0x5635bdf479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.711+0000 7f32590f9640 1 -- 172.21.15.23:0/2272858954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635bde34a20 con 0x5635bdf36390 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f32590f9640 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f324402c840 tx=0x7f324402cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c458242b5bffcf51 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f323f7fe640 1 -- 172.21.15.23:0/2272858954 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244004260 con 0x5635bdf36390 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f323f7fe640 1 -- 172.21.15.23:0/2272858954 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3244049040 con 0x5635bdf36390 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f323f7fe640 1 -- 172.21.15.23:0/2272858954 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32440051f0 con 0x5635bdf36390 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 -- 172.21.15.23:0/2272858954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 msgr2=0x5635bdf30e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f324402c840 tx=0x7f324402cc90 comp rx=0 tx=0).stop 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 -- 172.21.15.23:0/2272858954 shutdown_connections 2024-03-16T20:35:59.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf3f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 --2- 172.21.15.23:0/2272858954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf30e60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 -- 172.21.15.23:0/2272858954 >> 172.21.15.23:0/2272858954 conn(0x5635bdf2d4f0 msgr2=0x5635bdf30750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 -- 172.21.15.23:0/2272858954 shutdown_connections 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.712+0000 7f325e81a740 1 -- 172.21.15.23:0/2272858954 wait complete. 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 -- start start 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 Processor -- start 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf44380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf448c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdf44fa0 con 0x5635bdf36390 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdfa7dd0 con 0x5635bdf36d80 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635bdfa7f60 con 0x5635bdf3bef0 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f325e81a740 10 client.0 _ll_register_callbacks cb 0x7f3250004f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:59.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38896/0 (socket says 172.21.15.23:38896) 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 -- 172.21.15.23:0/127677630 learned_addr learned my addr 172.21.15.23:0/127677630 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32598fa640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf44380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 -- 172.21.15.23:0/127677630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 msgr2=0x5635bdf448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 -- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 msgr2=0x5635bdf44380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf44380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.713+0000 7f32590f9640 1 -- 172.21.15.23:0/127677630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325410a240 con 0x5635bdf36390 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f32590f9640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3244036040 tx=0x7f3244005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37caf123fa0fdcef server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32440052f0 con 0x5635bdf36390 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3244005450 con 0x5635bdf36390 2024-03-16T20:35:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635bdf4ca80 con 0x5635bdf36390 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3244005610 con 0x5635bdf36390 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5635bde5a780 con 0x5635bdf36390 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5635bdf4ca80 con 0x5635bdf36390 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f324403c040 con 0x5635bdf36390 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 10 client.15459 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 20 client.15459 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdface70 0x5635bdfa9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 4 client.15459 mds_command: new command op to 24416 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.714+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 --> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3250004390 con 0x5635bdface70 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f32598fa640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdface70 0x5635bdfa9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f32598fa640 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdface70 0x5635bdfa9340 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87b238ceec98af92 server_cookie=8f546b1da1396d97 in_seq=0 out_seq=0 2024-03-16T20:35:59.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f323d7fa640 10 client.15459 ms_handle_connect on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f323d7fa640 1 -- 172.21.15.23:0/127677630 <== mds.4 v2:172.21.15.31:6832/1919282692 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7f3250004390 con 0x5635bdface70 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f323d7fa640 10 client.15459 handle_command_reply: tid=0 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 client.15459 shutdown 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 msgr2=0x5635bdf478e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3244036040 tx=0x7f3244005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 shutdown_connections 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5635bdf3bef0 0x5635bdf448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:6832/1919282692,v1:172.21.15.31:6833/1919282692] conn(0x5635bdface70 0x5635bdfa9340 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5635bdf36d80 0x5635bdf44380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 --2- 172.21.15.23:0/127677630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5635bdf36390 0x5635bdf478e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f323d7fa640 0 client.15459 ms_handle_reset on v2:172.21.15.31:6832/1919282692 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 >> 172.21.15.23:0/127677630 conn(0x5635bdf2d4f0 msgr2=0x5635bdf30fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 shutdown_connections 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.715+0000 7f325e81a740 1 -- 172.21.15.23:0/127677630 wait complete. 2024-03-16T20:35:59.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325e81a740 20 client.15459 trim_cache size 0 max 16384 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24416, 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 109.783510069, 2024-03-16T20:35:59.728 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:59.729 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:59.729 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:59.729 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 163.380866217 2024-03-16T20:35:59.729 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 msgr2=0x7f32400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 0x7f32400a3ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f324c096d20 tx=0x7f324c069cf0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 msgr2=0x7f325419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f324c00eb10 tx=0x7f324c00c2d0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 shutdown_connections 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f32400a1a30 0x7f32400a3ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32541030b0 0x7f325419eb50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32541026f0 0x7f3254197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 --2- 172.21.15.23:0/1196502196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3254101500 0x7f32541975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 >> 172.21.15.23:0/1196502196 conn(0x7f32540fca90 msgr2=0x7f32541059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 shutdown_connections 2024-03-16T20:35:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.716+0000 7f325bb5e640 1 -- 172.21.15.23:0/1196502196 wait complete. 2024-03-16T20:35:59.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-16T20:35:59.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 -- 172.21.15.23:0/216513556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01026f0 msgr2=0x7f99f0102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 --2- 172.21.15.23:0/216513556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01026f0 0x7f99f0102b70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009900 tx=0x7f99e002bc30 comp rx=0 tx=0).stop 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 -- 172.21.15.23:0/216513556 shutdown_connections 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 --2- 172.21.15.23:0/216513556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f01030b0 0x7f99f0109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 --2- 172.21.15.23:0/216513556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01026f0 0x7f99f0102b70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 --2- 172.21.15.23:0/216513556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f0101500 0x7f99f0101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 -- 172.21.15.23:0/216513556 >> 172.21.15.23:0/216513556 conn(0x7f99f00fca90 msgr2=0x7f99f00feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 -- 172.21.15.23:0/216513556 shutdown_connections 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.830+0000 7f99f758c640 1 -- 172.21.15.23:0/216513556 wait complete. 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 Processor -- start 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 -- start start 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f0101500 0x7f99f0197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f01026f0 0x7f99f0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f019f070 con 0x7f99f01030b0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f019f1e0 con 0x7f99f0101500 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f019f3a0 con 0x7f99f01026f0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38924/0 (socket says 172.21.15.23:38924) 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 -- 172.21.15.23:0/3789734457 learned_addr learned my addr 172.21.15.23:0/3789734457 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f01026f0 0x7f99f0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5328640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f0101500 0x7f99f0197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 -- 172.21.15.23:0/3789734457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f01026f0 msgr2=0x7f99f0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f01026f0 0x7f99f0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 -- 172.21.15.23:0/3789734457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f0101500 msgr2=0x7f99f0197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f0101500 0x7f99f0197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 -- 172.21.15.23:0/3789734457 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0009560 con 0x7f99f01030b0 2024-03-16T20:35:59.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f5b29640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f99e800eb10 tx=0x7f99e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99e67fc640 1 -- 172.21.15.23:0/3789734457 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8004b20 con 0x7f99f01030b0 2024-03-16T20:35:59.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99e67fc640 1 -- 172.21.15.23:0/3789734457 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99e8004c80 con 0x7f99f01030b0 2024-03-16T20:35:59.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f019f680 con 0x7f99f01030b0 2024-03-16T20:35:59.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.831+0000 7f99e67fc640 1 -- 172.21.15.23:0/3789734457 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8010650 con 0x7f99f01030b0 2024-03-16T20:35:59.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.832+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f019fb10 con 0x7f99f01030b0 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99e67fc640 1 -- 172.21.15.23:0/3789734457 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f99e8005020 con 0x7f99f01030b0 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 Processor -- start 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 -- start start 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f5d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f5ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f13cd0 con 0x564407f57f80 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99e67fc640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 0x7f99dc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407ec6f20 con 0x564407f58970 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99fa16d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407ec4450 con 0x564407f5dae0 2024-03-16T20:35:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.834+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38942/0 (socket says 172.21.15.23:38942) 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 -- 172.21.15.23:0/3381110057 learned_addr learned my addr 172.21.15.23:0/3381110057 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f5328640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f5d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f5b29640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 0x7f99dc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f5ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99e67fc640 1 -- 172.21.15.23:0/3789734457 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f99e8014070 con 0x7f99f01030b0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 -- 172.21.15.23:0/3381110057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 msgr2=0x564407f5ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f5ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 -- 172.21.15.23:0/3381110057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 msgr2=0x564407f5d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f5d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 -- 172.21.15.23:0/3381110057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564407e74f20 con 0x564407f57f80 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f5328640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f5d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f5b29640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 0x7f99dc0a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99e8096d10 tx=0x7f99e800cf70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f99e002c170 tx=0x7f99e0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca6b14914ea47fa server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99db7fe640 1 -- 172.21.15.23:0/3381110057 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e00046c0 con 0x564407f57f80 2024-03-16T20:35:59.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99db7fe640 1 -- 172.21.15.23:0/3381110057 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99e0004510 con 0x564407f57f80 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.835+0000 7f99db7fe640 1 -- 172.21.15.23:0/3381110057 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0005290 con 0x564407f57f80 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- 172.21.15.23:0/3381110057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 msgr2=0x564407f583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f99e002c170 tx=0x7f99e0005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- 172.21.15.23:0/3381110057 shutdown_connections 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f5ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f5d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3381110057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f583a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- 172.21.15.23:0/3381110057 >> 172.21.15.23:0/3381110057 conn(0x564407f534f0 msgr2=0x564407f538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- 172.21.15.23:0/3381110057 shutdown_connections 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- 172.21.15.23:0/3381110057 wait complete. 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 -- start start 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.836+0000 7f99fa16d740 1 Processor -- start 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f57f80 0x564407f621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f69720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407ebfc30 con 0x564407f58970 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407e64d30 con 0x564407f5dae0 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407b93310 con 0x564407f57f80 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99fa16d740 10 client.0 _ll_register_callbacks cb 0x564407e6cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38944/0 (socket says 172.21.15.23:38944) 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 -- 172.21.15.23:0/3569702647 learned_addr learned my addr 172.21.15.23:0/3569702647 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f57f80 0x564407f621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f69720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 -- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f57f80 msgr2=0x564407f621a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f57f80 0x564407f621a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 -- 172.21.15.23:0/3569702647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 msgr2=0x564407f69720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f69720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 -- 172.21.15.23:0/3569702647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f010a240 con 0x564407f58970 2024-03-16T20:35:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99f5328640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00ba40 tx=0x7f99ec00bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade152e1908e193b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.837+0000 7f99d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec016070 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f99ec003e80 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564407bbf6c0 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99ec011440 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564407ebe2b0 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564407bbf6c0 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f99ec01b7f0 con 0x564407f58970 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 10 client.15471 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 20 client.15471 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407f6e5f0 0x564407f6dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 4 client.15471 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 --> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564407c10270 con 0x564407f6e5f0 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.838+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407f6e5f0 0x564407f6dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99d8ff9640 10 client.15471 ms_handle_connect on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99f4b27640 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407f6e5f0 0x564407f6dea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eeae278c1c97d96e server_cookie=2bb730d2e0f0c622 in_seq=0 out_seq=0 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99d8ff9640 1 -- 172.21.15.23:0/3569702647 <== mds.3 v2:172.21.15.189:6832/1083314774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f99e005a4b0 con 0x564407f6e5f0 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99d8ff9640 10 client.15471 handle_command_reply: tid=0 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99fa16d740 1 client.15471 shutdown 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 msgr2=0x564407f626e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00ba40 tx=0x7f99ec00bf00 comp rx=0 tx=0).stop 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 shutdown_connections 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.839+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407f6e5f0 0x564407f6dea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f69720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f58970 0x564407f626e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99d8ff9640 0 client.15471 ms_handle_reset on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 --2- 172.21.15.23:0/3569702647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f57f80 0x564407f621a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 >> 172.21.15.23:0/3569702647 conn(0x564407f534f0 msgr2=0x564407f527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 shutdown_connections 2024-03-16T20:35:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 1 -- 172.21.15.23:0/3569702647 wait complete. 2024-03-16T20:35:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.840+0000 7f99fa16d740 20 client.15471 trim_cache size 0 max 16384 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.856+0000 7f99fa16d740 1 Processor -- start 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.856+0000 7f99fa16d740 1 -- start start 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 0x564407f696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407e45490 con 0x564407f57f80 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f60630 con 0x564407f5dae0 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99fa16d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f544e0 con 0x564407f58970 2024-03-16T20:35:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38948/0 (socket says 172.21.15.23:38948) 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 -- 172.21.15.23:0/779450661 learned_addr learned my addr 172.21.15.23:0/779450661 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f5328640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 0x564407f696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 -- 172.21.15.23:0/779450661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 msgr2=0x564407f696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 0x564407f696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 -- 172.21.15.23:0/779450661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 msgr2=0x564407f538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 -- 172.21.15.23:0/779450661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564407e5aa20 con 0x564407f57f80 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f5328640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 0x564407f696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:35:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99f4b27640 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005c40 tx=0x7f99e0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16cb16d283977d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99daffd640 1 -- 172.21.15.23:0/779450661 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0005d50 con 0x564407f57f80 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99daffd640 1 -- 172.21.15.23:0/779450661 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99e0055040 con 0x564407f57f80 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.857+0000 7f99daffd640 1 -- 172.21.15.23:0/779450661 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0042880 con 0x564407f57f80 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- 172.21.15.23:0/779450661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 msgr2=0x564407f5fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005c40 tx=0x7f99e0005fd0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- 172.21.15.23:0/779450661 shutdown_connections 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f5dae0 0x564407f538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f58970 0x564407f696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 --2- 172.21.15.23:0/779450661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f5fed0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- 172.21.15.23:0/779450661 >> 172.21.15.23:0/779450661 conn(0x564407f534f0 msgr2=0x564407f59390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- 172.21.15.23:0/779450661 shutdown_connections 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- 172.21.15.23:0/779450661 wait complete. 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 -- start start 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.858+0000 7f99fa16d740 1 Processor -- start 2024-03-16T20:35:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f65e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f68d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f69270 con 0x564407f57f80 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f693e0 con 0x564407f58970 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564407f695a0 con 0x564407f5dae0 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 10 client.0 _ll_register_callbacks cb 0x7f99ec0042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38962/0 (socket says 172.21.15.23:38962) 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 -- 172.21.15.23:0/706148406 learned_addr learned my addr 172.21.15.23:0/706148406 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f5328640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f65e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f68d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 -- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 msgr2=0x564407f68d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f68d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 -- 172.21.15.23:0/706148406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 msgr2=0x564407f65e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f65e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 -- 172.21.15.23:0/706148406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564407e74f20 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f5328640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f65e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99f4b27640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005940 tx=0x7f99e002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13ad51cd6133af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0042c30 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f99e0042d90 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564407f60f70 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99e0004eb0 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.859+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564407e80780 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564407f60f70 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f99e003c030 con 0x564407f57f80 2024-03-16T20:35:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 10 client.15477 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 20 client.15477 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407fce380 0x564407fca860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 4 client.15477 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 --> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564407f6ef80 con 0x564407fce380 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99f5328640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407fce380 0x564407fca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99d8ff9640 10 client.15477 ms_handle_connect on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99f5328640 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407fce380 0x564407fca860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be1dce9e355ba585 server_cookie=a6cca4606f479dae in_seq=0 out_seq=0 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99d8ff9640 1 -- 172.21.15.23:0/706148406 <== mds.3 v2:172.21.15.189:6832/1083314774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x564407f6ef80 con 0x564407fce380 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99d8ff9640 10 client.15477 handle_command_reply: tid=0 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.860+0000 7f99fa16d740 1 client.15477 shutdown 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 msgr2=0x564407f65910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005940 tx=0x7f99e002c7e0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 shutdown_connections 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:6832/1083314774,v1:172.21.15.189:6833/1083314774] conn(0x564407fce380 0x564407fca860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564407f5dae0 0x564407f68d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99d8ff9640 0 client.15477 ms_handle_reset on v2:172.21.15.189:6832/1083314774 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564407f58970 0x564407f65e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 --2- 172.21.15.23:0/706148406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564407f57f80 0x564407f65910 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 >> 172.21.15.23:0/706148406 conn(0x564407f534f0 msgr2=0x564407f538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 shutdown_connections 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 1 -- 172.21.15.23:0/706148406 wait complete. 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99fa16d740 20 client.15477 trim_cache size 0 max 16384 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:35:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24418, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 110.93687251, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 166.327791641 2024-03-16T20:35:59.874 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:35:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 msgr2=0x7f99dc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 0x7f99dc0a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99e8096d10 tx=0x7f99e800cf70 comp rx=0 tx=0).stop 2024-03-16T20:35:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 msgr2=0x7f99f019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f99e800eb10 tx=0x7f99e800c2d0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 shutdown_connections 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f99dc0a1a30 0x7f99dc0a3ef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99f01030b0 0x7f99f019eb30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f01026f0 0x7f99f0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 --2- 172.21.15.23:0/3789734457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f0101500 0x7f99f0197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 >> 172.21.15.23:0/3789734457 conn(0x7f99f00fca90 msgr2=0x7f99f0105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 shutdown_connections 2024-03-16T20:35:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.861+0000 7f99f758c640 1 -- 172.21.15.23:0/3789734457 wait complete. 2024-03-16T20:35:59.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 -- 172.21.15.23:0/2308294340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988101fc0 msgr2=0x7f7988108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 --2- 172.21.15.23:0/2308294340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988101fc0 0x7f7988108850 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ca0 tx=0x7f798002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 -- 172.21.15.23:0/2308294340 shutdown_connections 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 --2- 172.21.15.23:0/2308294340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988101fc0 0x7f7988108850 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 --2- 172.21.15.23:0/2308294340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101600 0x7f7988101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 --2- 172.21.15.23:0/2308294340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988100410 0x7f7988100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 -- 172.21.15.23:0/2308294340 >> 172.21.15.23:0/2308294340 conn(0x7f79880fb9a0 msgr2=0x7f79880fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 -- 172.21.15.23:0/2308294340 shutdown_connections 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.976+0000 7f799069a640 1 -- 172.21.15.23:0/2308294340 wait complete. 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 Processor -- start 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 -- start start 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988101600 0x7f7988197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101fc0 0x7f798819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819f060 con 0x7f7988100410 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819f1d0 con 0x7f7988101fc0 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f799069a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819f390 con 0x7f7988101600 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38974/0 (socket says 172.21.15.23:38974) 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 -- 172.21.15.23:0/3133915313 learned_addr learned my addr 172.21.15.23:0/3133915313 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798ec37640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101fc0 0x7f798819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798dc35640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988101600 0x7f7988197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 -- 172.21.15.23:0/3133915313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988101600 msgr2=0x7f7988197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988101600 0x7f7988197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 -- 172.21.15.23:0/3133915313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101fc0 msgr2=0x7f798819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101fc0 0x7f798819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 -- 172.21.15.23:0/3133915313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c009490 con 0x7f7988100410 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.977+0000 7f798e436640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f797800e9c0 tx=0x7f797800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.978+0000 7f79877fe640 1 -- 172.21.15.23:0/3133915313 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780096e0 con 0x7f7988100410 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.978+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7980009900 con 0x7f7988100410 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.978+0000 7f79877fe640 1 -- 172.21.15.23:0/3133915313 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f797800cd80 con 0x7f7988100410 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.978+0000 7f79877fe640 1 -- 172.21.15.23:0/3133915313 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978010430 con 0x7f7988100410 2024-03-16T20:35:59.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.978+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798819f9a0 con 0x7f7988100410 2024-03-16T20:35:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f79877fe640 1 -- 172.21.15.23:0/3133915313 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7978003680 con 0x7f7988100410 2024-03-16T20:35:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 Processor -- start 2024-03-16T20:35:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 -- start start 2024-03-16T20:35:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f41380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3ef0cd0 con 0x5631d3f39d80 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3ea3f20 con 0x5631d3f39390 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f7993276740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3ea1450 con 0x5631d3f3eef0 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.980+0000 7f798ec37640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38982/0 (socket says 172.21.15.23:38982) 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798e436640 1 -- 172.21.15.23:0/566203074 learned_addr learned my addr 172.21.15.23:0/566203074 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798dc35640 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f41380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f79877fe640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 0x7f79700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 -- 172.21.15.23:0/566203074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 msgr2=0x5631d3f41380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f41380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 -- 172.21.15.23:0/566203074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 msgr2=0x5631d3f3e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 -- 172.21.15.23:0/566203074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631d3e51f20 con 0x5631d3f39390 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798e436640 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f79877fe640 1 -- 172.21.15.23:0/3133915313 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f7978014070 con 0x7f7988100410 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 0x7f79700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f798ec37640 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f397b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f798002c630 tx=0x7f7980005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69364a9331dc7ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f7984ff9640 1 -- 172.21.15.23:0/566203074 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800329b0 con 0x5631d3f39390 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f7984ff9640 1 -- 172.21.15.23:0/566203074 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7980034780 con 0x5631d3f39390 2024-03-16T20:35:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.981+0000 7f7984ff9640 1 -- 172.21.15.23:0/566203074 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800042a0 con 0x5631d3f39390 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- 172.21.15.23:0/566203074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 msgr2=0x5631d3f397b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f798ec37640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 0x7f79700a3ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7980049000 tx=0x7f798004a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f397b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f798002c630 tx=0x7f7980005f70 comp rx=0 tx=0).stop 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- 172.21.15.23:0/566203074 shutdown_connections 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f41380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f3e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 --2- 172.21.15.23:0/566203074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f397b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- 172.21.15.23:0/566203074 >> 172.21.15.23:0/566203074 conn(0x5631d3f304f0 msgr2=0x5631d3f308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- 172.21.15.23:0/566203074 shutdown_connections 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- 172.21.15.23:0/566203074 wait complete. 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.982+0000 7f7993276740 1 -- start start 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 Processor -- start 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f43580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f4ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3e9cc30 con 0x5631d3f39d80 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3e41d30 con 0x5631d3f39390 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3b70310 con 0x5631d3f3eef0 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f7993276740 10 client.0 _ll_register_callbacks cb 0x5631d3e49d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38984/0 (socket says 172.21.15.23:38984) 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 -- 172.21.15.23:0/2866287354 learned_addr learned my addr 172.21.15.23:0/2866287354 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798e436640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f43580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798ec37640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f4ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 -- 172.21.15.23:0/2866287354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 msgr2=0x5631d3f4ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f4ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 -- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 msgr2=0x5631d3f43580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:35:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f43580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 -- 172.21.15.23:0/2866287354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7988109150 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f798dc35640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f797c0128b0 tx=0x7f797c012cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3badf2f9c39eeac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.983+0000 7f796dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c017070 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f797c00f8c0 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631d3b9c6c0 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f797c0122e0 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5631d3e9b2b0 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5631d3b9c6c0 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f797c010d00 con 0x5631d3f39d80 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 10 client.15489 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 20 client.15489 populate_metadata read hostname 'smithi023' 2024-03-16T20:35:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3f50090 0x5631d3f4f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 4 client.15489 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 --> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631d3bed270 con 0x5631d3f50090 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.984+0000 7f798e436640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3f50090 0x5631d3f4f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.985+0000 7f796dffb640 10 client.15489 ms_handle_connect on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.985+0000 7f798e436640 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3f50090 0x5631d3f4f280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca2f5c242dda6e49 server_cookie=b27c3389a6490684 in_seq=0 out_seq=0 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f796dffb640 1 -- 172.21.15.23:0/2866287354 <== mds.2 v2:172.21.15.31:6836/424329834 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5631d3bed270 con 0x5631d3f50090 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f796dffb640 10 client.15489 handle_command_reply: tid=0 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 client.15489 shutdown 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 msgr2=0x5631d3f43ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f797c0128b0 tx=0x7f797c012cb0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 shutdown_connections 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3f50090 0x5631d3f4f280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f4ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f796dffb640 0 client.15489 ms_handle_reset on v2:172.21.15.31:6836/424329834 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f43ac0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 --2- 172.21.15.23:0/2866287354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39390 0x5631d3f43580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 >> 172.21.15.23:0/2866287354 conn(0x5631d3f304f0 msgr2=0x5631d3f2f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 shutdown_connections 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 1 -- 172.21.15.23:0/2866287354 wait complete. 2024-03-16T20:35:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:35:59.986+0000 7f7993276740 20 client.15489 trim_cache size 0 max 16384 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 Processor -- start 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 -- start start 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f39390 0x5631d3f43a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f3eef0 0x5631d3f2b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3e22490 con 0x5631d3f39d80 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3f52070 con 0x5631d3f3eef0 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f7993276740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3bed270 con 0x5631d3f39390 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38994/0 (socket says 172.21.15.23:38994) 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 -- 172.21.15.23:0/3292165686 learned_addr learned my addr 172.21.15.23:0/3292165686 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798e436640 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f39390 0x5631d3f43a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798ec37640 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f3eef0 0x5631d3f2b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 -- 172.21.15.23:0/3292165686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f39390 msgr2=0x5631d3f43a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f39390 0x5631d3f43a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 -- 172.21.15.23:0/3292165686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f3eef0 msgr2=0x5631d3f2b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f3eef0 0x5631d3f2b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.003+0000 7f798dc35640 1 -- 172.21.15.23:0/3292165686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631d3e37a20 con 0x5631d3f39d80 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f798dc35640 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f797c0059f0 tx=0x7f797c0122e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b2323e133c1128 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f796ffff640 1 -- 172.21.15.23:0/3292165686 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c017040 con 0x5631d3f39d80 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f796ffff640 1 -- 172.21.15.23:0/3292165686 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f797c01ce90 con 0x5631d3f39d80 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f796ffff640 1 -- 172.21.15.23:0/3292165686 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c003850 con 0x5631d3f39d80 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 -- 172.21.15.23:0/3292165686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 msgr2=0x5631d3f4aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f797c0059f0 tx=0x7f797c0122e0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 -- 172.21.15.23:0/3292165686 shutdown_connections 2024-03-16T20:36:00.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f3eef0 0x5631d3f2b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39d80 0x5631d3f4aac0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 --2- 172.21.15.23:0/3292165686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f39390 0x5631d3f43a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 -- 172.21.15.23:0/3292165686 >> 172.21.15.23:0/3292165686 conn(0x5631d3f304f0 msgr2=0x5631d3f308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 -- 172.21.15.23:0/3292165686 shutdown_connections 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.004+0000 7f7993276740 1 -- 172.21.15.23:0/3292165686 wait complete. 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 -- start start 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 Processor -- start 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 0x5631d3f4aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f47100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3f47850 con 0x5631d3f39390 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3f479c0 con 0x5631d3f39d80 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631d3f47b80 con 0x5631d3f3eef0 2024-03-16T20:36:00.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f7993276740 10 client.0 _ll_register_callbacks cb 0x7f797c0099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38996/0 (socket says 172.21.15.23:38996) 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 -- 172.21.15.23:0/667841820 learned_addr learned my addr 172.21.15.23:0/667841820 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798dc35640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 0x5631d3f4aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798ec37640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f47100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 -- 172.21.15.23:0/667841820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 msgr2=0x5631d3f47100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f47100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 -- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 msgr2=0x5631d3f4aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 0x5631d3f4aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798e436640 1 -- 172.21.15.23:0/667841820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631d3e9b2b0 con 0x5631d3f39390 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.005+0000 7f798dc35640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 0x5631d3f4aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f798e436640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f797809a690 tx=0x7f79780cb160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce6e2c6c0cc317b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:36:00.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797809ae00 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f79780a2ea0 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7980034780 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f797809a700 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5631d3e5d780 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79780b8020 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f79780d5040 con 0x5631d3f39390 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 10 client.15495 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 20 client.15495 populate_metadata read hostname 'smithi023' 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3fb0190 0x5631d3fac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 4 client.15495 mds_command: new command op to 24493 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 --> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5631d3bd9610 con 0x5631d3fb0190 2024-03-16T20:36:00.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.006+0000 7f798dc35640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3fb0190 0x5631d3fac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f796dffb640 10 client.15495 ms_handle_connect on v2:172.21.15.31:6836/424329834 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f798dc35640 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3fb0190 0x5631d3fac5b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10c87a76f09e9e44 server_cookie=eeee8a2ca1ac7d95 in_seq=0 out_seq=0 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f796dffb640 1 -- 172.21.15.23:0/667841820 <== mds.2 v2:172.21.15.31:6836/424329834 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5631d3bd9610 con 0x5631d3fb0190 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f796dffb640 10 client.15495 handle_command_reply: tid=0 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 client.15495 shutdown 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 msgr2=0x5631d3f4a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f797809a690 tx=0x7f79780cb160 comp rx=0 tx=0).stop 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 shutdown_connections 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:6836/424329834,v1:172.21.15.31:6837/424329834] conn(0x5631d3fb0190 0x5631d3fac5b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631d3f3eef0 0x5631d3f47100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631d3f39d80 0x5631d3f4aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 --2- 172.21.15.23:0/667841820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631d3f39390 0x5631d3f4a4d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f796dffb640 0 client.15495 ms_handle_reset on v2:172.21.15.31:6836/424329834 2024-03-16T20:36:00.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 >> 172.21.15.23:0/667841820 conn(0x5631d3f304f0 msgr2=0x5631d3f43170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 shutdown_connections 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 1 -- 172.21.15.23:0/667841820 wait complete. 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.007+0000 7f7993276740 20 client.15495 trim_cache size 0 max 16384 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24493, 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:36:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 111.290204777, 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 146.828391243 2024-03-16T20:36:00.020 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 msgr2=0x7f79700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 0x7f79700a3ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7980049000 tx=0x7f798004a000 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 msgr2=0x7f79881975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f797800e9c0 tx=0x7f797800ee80 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 shutdown_connections 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79700a1a30 0x7f79700a3ef0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7988101fc0 0x7f798819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7988101600 0x7f7988197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 --2- 172.21.15.23:0/3133915313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7988100410 0x7f79881975c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 >> 172.21.15.23:0/3133915313 conn(0x7f79880fb9a0 msgr2=0x7f79880fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 shutdown_connections 2024-03-16T20:36:00.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.008+0000 7f799069a640 1 -- 172.21.15.23:0/3133915313 wait complete. 2024-03-16T20:36:00.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 -- 172.21.15.23:0/299991413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c0ffb90 msgr2=0x7f427c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 --2- 172.21.15.23:0/299991413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c0ffb90 0x7f427c0fffb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4270009900 tx=0x7f427002bc30 comp rx=0 tx=0).stop 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 -- 172.21.15.23:0/299991413 shutdown_connections 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 --2- 172.21.15.23:0/299991413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c101740 0x7f427c107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 --2- 172.21.15.23:0/299991413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c100d80 0x7f427c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 --2- 172.21.15.23:0/299991413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c0ffb90 0x7f427c0fffb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 -- 172.21.15.23:0/299991413 >> 172.21.15.23:0/299991413 conn(0x7f427c0fb100 msgr2=0x7f427c0fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 -- 172.21.15.23:0/299991413 shutdown_connections 2024-03-16T20:36:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.122+0000 7f4282bc7640 1 -- 172.21.15.23:0/299991413 wait complete. 2024-03-16T20:36:00.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 Processor -- start 2024-03-16T20:36:00.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 -- start start 2024-03-16T20:36:00.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c0ffb90 0x7f427c109440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 0x7f427c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c19f120 con 0x7f427c101740 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c19f290 con 0x7f427c100d80 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4282bc7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c19f450 con 0x7f427c0ffb90 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 0x7f427c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:33154/0 (socket says 172.21.15.23:33154) 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 0x7f427c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39002/0 (socket says 172.21.15.23:39002) 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 -- 172.21.15.23:0/972249804 learned_addr learned my addr 172.21.15.23:0/972249804 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f4280963640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c0ffb90 0x7f427c109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 -- 172.21.15.23:0/972249804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c0ffb90 msgr2=0x7f427c109440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c0ffb90 0x7f427c109440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 -- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 msgr2=0x7f427c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 0x7f427c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 -- 172.21.15.23:0/972249804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268009490 con 0x7f427c100d80 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.123+0000 7f427bfff640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f426800ea60 tx=0x7f426800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.124+0000 7f4279ffb640 1 -- 172.21.15.23:0/972249804 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426800cde0 con 0x7f427c100d80 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.124+0000 7f4279ffb640 1 -- 172.21.15.23:0/972249804 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42680035f0 con 0x7f427c100d80 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.124+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4270009560 con 0x7f427c100d80 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.124+0000 7f4279ffb640 1 -- 172.21.15.23:0/972249804 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268010690 con 0x7f427c100d80 2024-03-16T20:36:00.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.124+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c19fa50 con 0x7f427c100d80 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.125+0000 7f4279ffb640 1 -- 172.21.15.23:0/972249804 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4268003af0 con 0x7f427c100d80 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.126+0000 7f4279ffb640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 0x7f426c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.126+0000 7f4280963640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 0x7f426c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.126+0000 7f428579d740 1 Processor -- start 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.126+0000 7f4279ffb640 1 -- 172.21.15.23:0/972249804 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4268014070 con 0x7f427c100d80 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.126+0000 7f428579d740 1 -- start start 2024-03-16T20:36:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffee63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffeeb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffea1cd0 con 0x5603ffeebae0 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffe54f20 con 0x5603ffee5f80 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f428579d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffe52450 con 0x5603ffee6970 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4280963640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 0x7f426c0a3ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4270005ab0 tx=0x7f4270005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39008/0 (socket says 172.21.15.23:39008) 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 -- 172.21.15.23:0/1259396346 learned_addr learned my addr 172.21.15.23:0/1259396346 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f427bfff640 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffeeb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4280963640 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffee63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 -- 172.21.15.23:0/1259396346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 msgr2=0x5603ffeeb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffeeb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 -- 172.21.15.23:0/1259396346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 msgr2=0x5603ffee63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffee63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 -- 172.21.15.23:0/1259396346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603ffe02f20 con 0x5603ffeebae0 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4281164640 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4274009fd0 tx=0x7f42740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4321186a57109818 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4266ffd640 1 -- 172.21.15.23:0/1259396346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274004d20 con 0x5603ffeebae0 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4266ffd640 1 -- 172.21.15.23:0/1259396346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4274004e80 con 0x5603ffeebae0 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.127+0000 7f4266ffd640 1 -- 172.21.15.23:0/1259396346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274005280 con 0x5603ffeebae0 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- 172.21.15.23:0/1259396346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 msgr2=0x5603ffeedf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4274009fd0 tx=0x7f42740118d0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- 172.21.15.23:0/1259396346 shutdown_connections 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffeedf70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffeeb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 --2- 172.21.15.23:0/1259396346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffee63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- 172.21.15.23:0/1259396346 >> 172.21.15.23:0/1259396346 conn(0x5603ffee14f0 msgr2=0x5603ffee18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- 172.21.15.23:0/1259396346 shutdown_connections 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- 172.21.15.23:0/1259396346 wait complete. 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 -- start start 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.128+0000 7f428579d740 1 Processor -- start 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 0x5603ffef0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee6970 0x5603ffef06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffe4dc30 con 0x5603ffeebae0 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffdf2d30 con 0x5603ffee6970 2024-03-16T20:36:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffb21310 con 0x5603ffee5f80 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f428579d740 10 client.0 _ll_register_callbacks cb 0x5603ffdfad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4280963640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 0x5603ffef0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4280963640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 0x5603ffef0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:51058/0 (socket says 172.21.15.23:51058) 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4280963640 1 -- 172.21.15.23:0/3127805027 learned_addr learned my addr 172.21.15.23:0/3127805027 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39022/0 (socket says 172.21.15.23:39022) 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f427bfff640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee6970 0x5603ffef06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 -- 172.21.15.23:0/3127805027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 msgr2=0x5603ffef0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 0x5603ffef0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 -- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee6970 msgr2=0x5603ffef06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee6970 0x5603ffef06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 -- 172.21.15.23:0/3127805027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4274009c80 con 0x5603ffeebae0 2024-03-16T20:36:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f4281164640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4274011dd0 tx=0x7f4274003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e0620347047487 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.129+0000 7f425bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274004460 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f427400f4e0 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603ffb9e270 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4274003dd0 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5603ffe4c2b0 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5603ffb9e270 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4274021040 con 0x5603ffeebae0 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 10 client.15507 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T20:36:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 20 client.15507 populate_metadata read hostname 'smithi023' 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603ffefc880 0x5603fff00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 4 client.15507 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.130+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 --> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603ffb8a610 con 0x5603ffefc880 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.131+0000 7f4280963640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603ffefc880 0x5603fff00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.131+0000 7f4280963640 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603ffefc880 0x5603fff00d00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a5c13ded4c8ee4b8 server_cookie=43451cdb4fbfb3e7 in_seq=0 out_seq=0 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.131+0000 7f425bfff640 10 client.15507 ms_handle_connect on v2:172.21.15.31:6838/37663771 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f425bfff640 1 -- 172.21.15.23:0/3127805027 <== mds.1 v2:172.21.15.31:6838/37663771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5603ffb8a610 con 0x5603ffefc880 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f425bfff640 10 client.15507 handle_command_reply: tid=0 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 client.15507 shutdown 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 msgr2=0x5603ffef76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4274011dd0 tx=0x7f4274003c60 comp rx=0 tx=0).stop 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 shutdown_connections 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603ffefc880 0x5603fff00d00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffef76f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee6970 0x5603ffef06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f425bfff640 0 client.15507 ms_handle_reset on v2:172.21.15.31:6838/37663771 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 --2- 172.21.15.23:0/3127805027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee5f80 0x5603ffef0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 >> 172.21.15.23:0/3127805027 conn(0x5603ffee14f0 msgr2=0x5603ffee07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 shutdown_connections 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 1 -- 172.21.15.23:0/3127805027 wait complete. 2024-03-16T20:36:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.132+0000 7f428579d740 20 client.15507 trim_cache size 0 max 16384 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 Processor -- start 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 -- start start 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffeeb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffdd3490 con 0x5603ffeebae0 2024-03-16T20:36:00.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffb8a610 con 0x5603ffee5f80 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f428579d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffb9e270 con 0x5603ffee6970 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f4281164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39028/0 (socket says 172.21.15.23:39028) 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f4281164640 1 -- 172.21.15.23:0/4080954122 learned_addr learned my addr 172.21.15.23:0/4080954122 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f4280963640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffeeb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.149+0000 7f427bfff640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 -- 172.21.15.23:0/4080954122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 msgr2=0x5603ffef76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 -- 172.21.15.23:0/4080954122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 msgr2=0x5603ffeeb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffeeb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 -- 172.21.15.23:0/4080954122 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603ffde8a20 con 0x5603ffeebae0 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4280963640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffeeb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f4281164640 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4274011910 tx=0x7f42740056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fe13568158363b server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f42667fc640 1 -- 172.21.15.23:0/4080954122 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274011e00 con 0x5603ffeebae0 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f42667fc640 1 -- 172.21.15.23:0/4080954122 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4274009ea0 con 0x5603ffeebae0 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f42667fc640 1 -- 172.21.15.23:0/4080954122 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427400f4e0 con 0x5603ffeebae0 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 -- 172.21.15.23:0/4080954122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 msgr2=0x5603ffee4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4274011910 tx=0x7f42740056d0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 -- 172.21.15.23:0/4080954122 shutdown_connections 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffeebae0 0x5603ffee4e60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 --2- 172.21.15.23:0/4080954122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffee5f80 0x5603ffeeb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.150+0000 7f428579d740 1 -- 172.21.15.23:0/4080954122 >> 172.21.15.23:0/4080954122 conn(0x5603ffee14f0 msgr2=0x5603ffee4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- 172.21.15.23:0/4080954122 shutdown_connections 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- 172.21.15.23:0/4080954122 wait complete. 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- start start 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 Processor -- start 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffeebae0 0x5603ffef3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffef44a0 con 0x5603ffee5f80 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffef4610 con 0x5603ffeebae0 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603ffef47d0 con 0x5603ffee6970 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f428579d740 10 client.0 _ll_register_callbacks cb 0x7f4268005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f4280963640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f4280963640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39036/0 (socket says 172.21.15.23:39036) 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f4280963640 1 -- 172.21.15.23:0/176218785 learned_addr learned my addr 172.21.15.23:0/176218785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f427bfff640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.151+0000 7f4281164640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffeebae0 0x5603ffef3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 -- 172.21.15.23:0/176218785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 msgr2=0x5603ffef76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 -- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffeebae0 msgr2=0x5603ffef3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffeebae0 0x5603ffef3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 -- 172.21.15.23:0/176218785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c069250 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f4280963640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4270009230 tx=0x7f427002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4e34bea79a117c server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270053040 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f4270047040 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603ffefd250 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4270004250 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 10 client.15513 fetch_fsmap learned FSMap version 28 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5603ffe4c2b0 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5603ffefd250 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4270004e80 con 0x5603ffee5f80 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 10 client.15513 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-16T20:36:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 20 client.15513 populate_metadata read hostname 'smithi023' 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603fff5cef0 0x5603fff59330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 4 client.15513 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.152+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 --> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5603ffb4d6c0 con 0x5603fff5cef0 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f427bfff640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603fff5cef0 0x5603fff59330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f425bfff640 10 client.15513 ms_handle_connect on v2:172.21.15.31:6838/37663771 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f427bfff640 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603fff5cef0 0x5603fff59330 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d31057bbdfa1f955 server_cookie=aede7b70657d0095 in_seq=0 out_seq=0 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f425bfff640 1 -- 172.21.15.23:0/176218785 <== mds.1 v2:172.21.15.31:6838/37663771 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5603ffb4d6c0 con 0x5603fff5cef0 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f425bfff640 10 client.15513 handle_command_reply: tid=0 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 client.15513 shutdown 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 msgr2=0x5603ffef7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4270009230 tx=0x7f427002bc30 comp rx=0 tx=0).stop 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 shutdown_connections 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:6838/37663771,v1:172.21.15.31:6839/37663771] conn(0x5603fff5cef0 0x5603fff59330 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603ffeebae0 0x5603ffef3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603ffee6970 0x5603ffef76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 --2- 172.21.15.23:0/176218785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5603ffee5f80 0x5603ffef7190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f425bfff640 0 client.15513 ms_handle_reset on v2:172.21.15.31:6838/37663771 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 >> 172.21.15.23:0/176218785 conn(0x5603ffee14f0 msgr2=0x5603ffee4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 shutdown_connections 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.153+0000 7f428579d740 1 -- 172.21.15.23:0/176218785 wait complete. 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f428579d740 20 client.15513 trim_cache size 0 max 16384 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-16T20:36:00.165 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24497, 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 112.452050431, 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 135.504461263 2024-03-16T20:36:00.166 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 msgr2=0x7f426c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 0x7f426c0a3ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4270005ab0 tx=0x7f4270005fd0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 msgr2=0x7f427c19c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f426800ea60 tx=0x7f426800ef20 comp rx=0 tx=0).stop 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 shutdown_connections 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f426c0a1a30 0x7f426c0a3ef0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f427c101740 0x7f427c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f427c100d80 0x7f427c19c320 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 --2- 172.21.15.23:0/972249804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427c0ffb90 0x7f427c109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 >> 172.21.15.23:0/972249804 conn(0x7f427c0fb100 msgr2=0x7f427c104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 shutdown_connections 2024-03-16T20:36:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.154+0000 7f4282bc7640 1 -- 172.21.15.23:0/972249804 wait complete. 2024-03-16T20:36:00.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/214775755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1049765470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:00 smithi023 ceph-mon[31879]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:00.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-16T20:36:00.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.268+0000 7fd8f88ff640 1 -- 172.21.15.23:0/437523443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4100410 msgr2=0x7fd8f4100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.268+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/437523443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4100410 0x7fd8f4100830 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009900 tx=0x7fd8e402bc30 comp rx=0 tx=0).stop 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 -- 172.21.15.23:0/437523443 shutdown_connections 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/437523443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4101fc0 0x7fd8f4108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/437523443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101600 0x7fd8f4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/437523443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4100410 0x7fd8f4100830 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 -- 172.21.15.23:0/437523443 >> 172.21.15.23:0/437523443 conn(0x7fd8f40fb9a0 msgr2=0x7fd8f40fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 -- 172.21.15.23:0/437523443 shutdown_connections 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 -- 172.21.15.23:0/437523443 wait complete. 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 Processor -- start 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 -- start start 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 0x7fd8f406a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.269+0000 7fd8f88ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101fc0 0x7fd8f419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f88ff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f419f090 con 0x7fd8f4101600 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f88ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f419f200 con 0x7fd8f4101fc0 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f88ff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f419f3c0 con 0x7fd8f4100410 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39044/0 (socket says 172.21.15.23:39044) 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1727085196 learned_addr learned my addr 172.21.15.23:0/1727085196 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f259c640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 0x7fd8f406a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101fc0 0x7fd8f419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1727085196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 msgr2=0x7fd8f406a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 0x7fd8f406a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1727085196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101fc0 msgr2=0x7fd8f419e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101fc0 0x7fd8f419e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1727085196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4009560 con 0x7fd8f4101600 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f259c640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 0x7fd8f406a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9f0 tx=0x7fd8dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8eb7fe640 1 -- 172.21.15.23:0/1727085196 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc009700 con 0x7fd8f4101600 2024-03-16T20:36:00.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f419f6a0 con 0x7fd8f4101600 2024-03-16T20:36:00.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8eb7fe640 1 -- 172.21.15.23:0/1727085196 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8dc00cd80 con 0x7fd8f4101600 2024-03-16T20:36:00.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8eb7fe640 1 -- 172.21.15.23:0/1727085196 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc018920 con 0x7fd8f4101600 2024-03-16T20:36:00.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.270+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f419fc00 con 0x7fd8f4101600 2024-03-16T20:36:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.272+0000 7fd8eb7fe640 1 -- 172.21.15.23:0/1727085196 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd8dc018a80 con 0x7fd8f4101600 2024-03-16T20:36:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 Processor -- start 2024-03-16T20:36:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 -- start start 2024-03-16T20:36:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9838c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d98393f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d98347cd0 con 0x563d9838c970 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d982faf20 con 0x563d98391ae0 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d982f8450 con 0x563d9838bf80 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.273+0000 7fd8eb7fe640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 0x7fd8e00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39058/0 (socket says 172.21.15.23:39058) 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 -- 172.21.15.23:0/3901555850 learned_addr learned my addr 172.21.15.23:0/3901555850 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d98393f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9838c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 0x7fd8e00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8eb7fe640 1 -- 172.21.15.23:0/1727085196 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fd8dc014070 con 0x7fd8f4101600 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 -- 172.21.15.23:0/3901555850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 msgr2=0x563d9838c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9838c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 -- 172.21.15.23:0/3901555850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 msgr2=0x563d98393f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d98393f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 -- 172.21.15.23:0/3901555850 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d982a8f20 con 0x563d9838c970 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f259c640 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009900 tx=0x7fd8e402cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361090611aac19cb server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 0x7fd8e00a3ef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x563d9838d1d0 tx=0x7fd8ec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8e8ff9640 1 -- 172.21.15.23:0/3901555850 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e40046c0 con 0x563d9838c970 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8e8ff9640 1 -- 172.21.15.23:0/3901555850 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8e4004510 con 0x563d9838c970 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.274+0000 7fd8e8ff9640 1 -- 172.21.15.23:0/3901555850 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e40053b0 con 0x563d9838c970 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/3901555850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 msgr2=0x563d98391470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009900 tx=0x7fd8e402cd20 comp rx=0 tx=0).stop 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/3901555850 shutdown_connections 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d98393f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98391470 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/3901555850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9838c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/3901555850 >> 172.21.15.23:0/3901555850 conn(0x563d983874f0 msgr2=0x563d983878f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/3901555850 shutdown_connections 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/3901555850 wait complete. 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 -- start start 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.275+0000 7fd8fb5b5740 1 Processor -- start 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98396130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d9839d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d982f3c30 con 0x563d9838c970 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d98298d30 con 0x563d98391ae0 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d97fc7310 con 0x563d9838bf80 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 10 client.0 _ll_register_callbacks cb 0x563d982a0d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8f259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98396130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39070/0 (socket says 172.21.15.23:39070) 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1743605280 learned_addr learned my addr 172.21.15.23:0/1743605280 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d9839d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 msgr2=0x563d98396130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98396130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1743605280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 msgr2=0x563d9839d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d9839d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 -- 172.21.15.23:0/1743605280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f4109150 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9c0 tx=0x7fd8dc0660a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c510bd608b2db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc09e850 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fd8dc0a2ea0 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d97ff36c0 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.276+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8dc0ba0f0 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563d982f22b0 con 0x563d9838c970 2024-03-16T20:36:00.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563d97ff36c0 con 0x563d9838c970 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd8dc099c00 con 0x563d9838c970 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 10 client.15525 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 20 client.15525 populate_metadata read hostname 'smithi023' 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d983a2aa0 0x563d983a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 4 client.15525 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d98044270 con 0x563d983a2aa0 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8f259c640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d983a2aa0 0x563d983a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8d9ffb640 10 client.15525 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.277+0000 7fd8f259c640 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d983a2aa0 0x563d983a1d30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e0921155acf7cd server_cookie=a868bc21c68e401e in_seq=0 out_seq=0 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/1743605280 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d98044270 con 0x563d983a2aa0 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8d9ffb640 10 client.15525 handle_command_reply: tid=0 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 client.15525 shutdown 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 msgr2=0x563d98396670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9c0 tx=0x7fd8dc0660a0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 shutdown_connections 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d983a2aa0 0x563d983a1d30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d98391ae0 0x563d9839d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d9838c970 0x563d98396670 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8d9ffb640 0 client.15525 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1743605280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98396130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 >> 172.21.15.23:0/1743605280 conn(0x563d983874f0 msgr2=0x563d983867d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 shutdown_connections 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1743605280 wait complete. 2024-03-16T20:36:00.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.278+0000 7fd8fb5b5740 20 client.15525 trim_cache size 0 max 16384 2024-03-16T20:36:00.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 Processor -- start 2024-03-16T20:36:00.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 -- start start 2024-03-16T20:36:00.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98385f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d98279490 con 0x563d98391ae0 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d98387fe0 con 0x563d9838c970 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d98044270 con 0x563d9838bf80 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39084/0 (socket says 172.21.15.23:39084) 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 -- 172.21.15.23:0/1711300974 learned_addr learned my addr 172.21.15.23:0/1711300974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f259c640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98385f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 -- 172.21.15.23:0/1711300974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 msgr2=0x563d98385f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98385f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 -- 172.21.15.23:0/1711300974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 msgr2=0x563d9839d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8f2d9d640 1 -- 172.21.15.23:0/1711300974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d9828ea20 con 0x563d98391ae0 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.295+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec011ea0 tx=0x7fd8ec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294b2c5f67705fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8dbfff640 1 -- 172.21.15.23:0/1711300974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec025070 con 0x563d98391ae0 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8dbfff640 1 -- 172.21.15.23:0/1711300974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8ec00fcc0 con 0x563d98391ae0 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8dbfff640 1 -- 172.21.15.23:0/1711300974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec020cf0 con 0x563d98391ae0 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1711300974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 msgr2=0x563d98386480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec011ea0 tx=0x7fd8ec005bd0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1711300974 shutdown_connections 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d98386480 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/1711300974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d98385f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1711300974 >> 172.21.15.23:0/1711300974 conn(0x563d983874f0 msgr2=0x563d983878d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1711300974 shutdown_connections 2024-03-16T20:36:00.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.296+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/1711300974 wait complete. 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 -- start start 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 Processor -- start 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9839d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d983fdb00 con 0x563d98391ae0 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d983fdc40 con 0x563d9838c970 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d983fdd80 con 0x563d9838bf80 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8fb5b5740 10 client.0 _ll_register_callbacks cb 0x7fd8dc01df80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8f2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8f259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9839d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8f2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39096/0 (socket says 172.21.15.23:39096) 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:33268/0 (socket says 172.21.15.23:33268) 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 -- 172.21.15.23:0/2836253680 learned_addr learned my addr 172.21.15.23:0/2836253680 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8f259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9839d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:51124/0 (socket says 172.21.15.23:51124) 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 -- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 msgr2=0x563d9839d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9839d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 -- 172.21.15.23:0/2836253680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 msgr2=0x563d9839a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8ebfff640 1 -- 172.21.15.23:0/2836253680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d982f22b0 con 0x563d9838c970 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.297+0000 7fd8f2d9d640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8ebfff640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc097180 tx=0x7fd8dc010db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2b511acd07fd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0850b0 con 0x563d9838c970 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d983a3470 con 0x563d9838c970 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mon.2 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fd8dc0a2ea0 con 0x563d9838c970 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8dc09e850 con 0x563d9838c970 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 10 client.25301 fetch_fsmap learned FSMap version 28 2024-03-16T20:36:00.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fd8ec019040 con 0x563d9838c970 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mon.2 v2:172.21.15.31:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563d983a3470 con 0x563d9838c970 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mon.2 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd8dc099670 con 0x563d9838c970 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 10 client.25301 fetch_fsmap finished waiting for FSMap version 28 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 10 client.25301 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 20 client.25301 populate_metadata read hostname 'smithi023' 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d98402e30 0x563d983ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 4 client.25301 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.298+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 --> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd8e4004510 con 0x563d98402e30 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8f259c640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d98402e30 0x563d983ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8d9ffb640 10 client.25301 ms_handle_connect on v2:172.21.15.189:6838/2333060283 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8f259c640 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d98402e30 0x563d983ff2b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2bbe0cb8e5a85f9 server_cookie=b694e702f9441857 in_seq=0 out_seq=0 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8d9ffb640 1 -- 172.21.15.23:0/2836253680 <== mds.0 v2:172.21.15.189:6838/2333060283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7fd8e4004510 con 0x563d98402e30 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8d9ffb640 10 client.25301 handle_command_reply: tid=0 2024-03-16T20:36:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 client.25301 shutdown 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 msgr2=0x563d9839a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc097180 tx=0x7fd8dc010db0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 shutdown_connections 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:6838/2333060283,v1:172.21.15.189:6839/2333060283] conn(0x563d98402e30 0x563d983ff2b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d98391ae0 0x563d9839a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8d9ffb640 0 client.25301 ms_handle_reset on v2:172.21.15.189:6838/2333060283 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d9838c970 0x563d9839a180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 --2- 172.21.15.23:0/2836253680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d9838bf80 0x563d9839d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 >> 172.21.15.23:0/2836253680 conn(0x563d983874f0 msgr2=0x563d9838a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 shutdown_connections 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 1 -- 172.21.15.23:0/2836253680 wait complete. 2024-03-16T20:36:00.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.299+0000 7fd8fb5b5740 20 client.25301 trim_cache size 0 max 16384 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "3d249ea8-e3d3-11ee-95c9-87774f69a715", 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24511, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 130.08963748299999, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 144.33484464899999 2024-03-16T20:36:00.312 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 msgr2=0x7fd8e00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 0x7fd8e00a3ef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x563d9838d1d0 tx=0x7fd8ec009500 comp rx=0 tx=0).stop 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 msgr2=0x7fd8f419c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00e9f0 tx=0x7fd8dc00eeb0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 shutdown_connections 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd8e00a1a30 0x7fd8e00a3ef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4101fc0 0x7fd8f419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f4101600 0x7fd8f419c290 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 --2- 172.21.15.23:0/1727085196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8f4100410 0x7fd8f406a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 >> 172.21.15.23:0/1727085196 conn(0x7fd8f40fb9a0 msgr2=0x7fd8f4104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 shutdown_connections 2024-03-16T20:36:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:00.300+0000 7fd8f88ff640 1 -- 172.21.15.23:0/1727085196 wait complete. 2024-03-16T20:36:00.315 INFO:teuthology.run_tasks:Running task kclient... 2024-03-16T20:36:00.327 INFO:tasks.kclient:Mounting kernel clients... 2024-03-16T20:36:00.327 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-16T20:36:00.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:00.328 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-16T20:36:00.343 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-16T20:36:00.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:00.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-16T20:36:00.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:00.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-16T20:36:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/214775755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1049765470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:00 smithi031 ceph-mon[36281]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/214775755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1049765470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:36:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:00 smithi189 ceph-mon[36655]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:00.506 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-16T20:36:00.506 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-16T20:36:00.506 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:00.507 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-16T20:36:00.752 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:36:02.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:02 smithi023 ceph-mon[31879]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:02.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 -- 172.21.15.23:0/1990051048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981066f0 msgr2=0x7fb498106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:02.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/1990051048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981066f0 0x7fb498106ad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0099c0 tx=0x7fb48c009cd0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 -- 172.21.15.23:0/1990051048 shutdown_connections 2024-03-16T20:36:02.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/1990051048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981066f0 0x7fb498106ad0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/1990051048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981012d0 0x7fb49810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/1990051048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 0x7fb498100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 -- 172.21.15.23:0/1990051048 >> 172.21.15.23:0/1990051048 conn(0x7fb498076210 msgr2=0x7fb498076620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:02.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 -- 172.21.15.23:0/1990051048 shutdown_connections 2024-03-16T20:36:02.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.394+0000 7fb49ca5f700 1 -- 172.21.15.23:0/1990051048 wait complete. 2024-03-16T20:36:02.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 Processor -- start 2024-03-16T20:36:02.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 -- start start 2024-03-16T20:36:02.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 0x7fb4981984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:02.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981066f0 0x7fb49819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49819d3d0 con 0x7fb4981012d0 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49819d540 con 0x7fb498100910 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.395+0000 7fb49ca5f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49819d700 con 0x7fb4981066f0 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60206/0 (socket says 172.21.15.23:60206) 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 -- 172.21.15.23:0/907916367 learned_addr learned my addr 172.21.15.23:0/907916367 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb49659c700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 0x7fb4981984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb496d9d700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981066f0 0x7fb49819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 -- 172.21.15.23:0/907916367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981066f0 msgr2=0x7fb49819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981066f0 0x7fb49819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 -- 172.21.15.23:0/907916367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 msgr2=0x7fb4981984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:02.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 0x7fb4981984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 -- 172.21.15.23:0/907916367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0095f0 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb495d9b700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb48800e9a0 tx=0x7fb48800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488009790 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb488010430 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49819d9e0 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488018680 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.396+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4981a5240 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.397+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb498109c50 con 0x7fb4981012d0 2024-03-16T20:36:02.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.399+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb488010910 con 0x7fb4981012d0 2024-03-16T20:36:02.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.400+0000 7fb4877fe700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 0x7fb4800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:02.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.400+0000 7fb49659c700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 0x7fb4800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:02.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.400+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fb488014070 con 0x7fb4981012d0 2024-03-16T20:36:02.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.400+0000 7fb49659c700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 0x7fb4800a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb48c000c00 tx=0x7fb48c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:02.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.403+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb488096d10 con 0x7fb4981012d0 2024-03-16T20:36:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:02 smithi031 ceph-mon[36281]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:02.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:02 smithi189 ceph-mon[36655]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.653+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb498068ca0 con 0x7fb4981012d0 2024-03-16T20:36:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.654+0000 7fb4877fe700 1 -- 172.21.15.23:0/907916367 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb488026070 con 0x7fb4981012d0 2024-03-16T20:36:02.659 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-16T20:36:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 msgr2=0x7fb4800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 0x7fb4800a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb48c000c00 tx=0x7fb48c005fb0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 msgr2=0x7fb498198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb48800e9a0 tx=0x7fb48800ecb0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 shutdown_connections 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fb4800a1ac0 0x7fb4800a3f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4981066f0 0x7fb49819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb498100910 0x7fb4981984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 --2- 172.21.15.23:0/907916367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4981012d0 0x7fb498198a30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 >> 172.21.15.23:0/907916367 conn(0x7fb498076210 msgr2=0x7fb49810e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 shutdown_connections 2024-03-16T20:36:02.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:02.656+0000 7fb49ca5f700 1 -- 172.21.15.23:0/907916367 wait complete. 2024-03-16T20:36:03.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:03 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/907916367' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:36:03.365 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-16T20:36:03.365 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-16T20:36:03.365 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-16T20:36:03.365 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-16T20:36:03.365 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:03.366 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-16T20:36:03.366 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-16T20:36:03.366 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-16T20:36:03.366 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-16T20:36:03.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.366 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-16T20:36:03.383 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-16T20:36:03.383 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-16T20:36:03.384 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42111sec preferred_lft 42111sec 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-16T20:36:03.385 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-16T20:36:03.385 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-16T20:36:03.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-16T20:36:03.386 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:36:03.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:03 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/907916367' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:36:03.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:03 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/907916367' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-16T20:36:03.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.499 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-16T20:36:03.525 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-16T20:36:03.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.528 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-16T20:36:03.584 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-16T20:36:03.585 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-16T20:36:03.585 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-16T20:36:03.585 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-16T20:36:03.585 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-16T20:36:03.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.585 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:36:03.585 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:36:03.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-16T20:36:03.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-16T20:36:03.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-16T20:36:03.586 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:36:03.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.682 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-16T20:36:03.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.698 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-16T20:36:03.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.753 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:36:03.753 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:36:03.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T20:36:03.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-16T20:36:03.753 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:36:03.835 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-16T20:36:03.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-16T20:36:03.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-16T20:36:03.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-16T20:36:03.836 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:36:03.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:03.961 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-16T20:36:03.961 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-16T20:36:03.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-16T20:36:03.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-16T20:36:03.961 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-16T20:36:04.008 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T20:36:04.008 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:04.023 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-16T20:36:04.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-16T20:36:04.023 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:04.080 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-16T20:36:04.080 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-16T20:36:04.080 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/5fa01b57-dcf5-4fa0-963f-99b4c0e05eb9 2024-03-16T20:36:04.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/5fa01b57-dcf5-4fa0-963f-99b4c0e05eb9 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-16T20:36:04.167 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-16T20:36:04.168 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-16T20:36:04.196 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:04 smithi023 ceph-mon[31879]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:04.295 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-16T20:36:04.296 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-16T20:36:04.296 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:6789,172.21.15.31:6789,172.21.15.189:6789" 2024-03-16T20:36:04.296 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with new device syntax: 0@3d249ea8-e3d3-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/5fa01b57-dcf5-4fa0-963f-99b4c0e05eb9 2024-03-16T20:36:04.296 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync,name=0,key=0,mon_addr=172.21.15.23:6789/172.21.15.31:6789/172.21.15.189:6789" will pass to kernel 2024-03-16T20:36:04.296 INFO:teuthology.orchestra.run.smithi023.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-16T20:36:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: You just mounted a file system that supports labels which does not 2024-03-16T20:36:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-16T20:36:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: applications will generate AVC messages and not be allowed access to 2024-03-16T20:36:04.297 INFO:teuthology.orchestra.run.smithi023.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-16T20:36:04.299 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-16T20:36:04.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:04.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.370 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T20:36:04.427 INFO:teuthology.orchestra.run.smithi023.stdout:client15540 2024-03-16T20:36:04.427 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:36:04.427 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T20:36:04.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:04 smithi031 ceph-mon[36281]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:04.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:04 smithi189 ceph-mon[36655]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:04.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.486 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T20:36:04.543 INFO:teuthology.orchestra.run.smithi023.stdout:client15540 2024-03-16T20:36:04.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.543 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-16T20:36:04.599 INFO:teuthology.orchestra.run.smithi023.stdout:client15540 2024-03-16T20:36:04.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:36:04.600 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715.client15540/status') 2024-03-16T20:36:04.666 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.15540 (3)192.168.0.1:0/1360993232 2024-03-16T20:36:04.667 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2024-03-16T20:36:04.667 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2024-03-16T20:36:04.667 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2024-03-16T20:36:04.667 INFO:teuthology.orchestra.run.smithi023.stderr:70 bytes copied, 2.0873e-05 s, 3.4 MB/s 2024-03-16T20:36:04.668 INFO:teuthology.run_tasks:Running task exec... 2024-03-16T20:36:04.678 INFO:teuthology.task.exec:Executing custom commands... 2024-03-16T20:36:04.678 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:36:04.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-16T20:36:04.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- 172.21.15.23:0/3953045344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 msgr2=0x7fbe40101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3953045344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40101200 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009900 tx=0x7fbe3402bc30 comp rx=0 tx=0).stop 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- 172.21.15.23:0/3953045344 shutdown_connections 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3953045344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 0x7fbe40107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3953045344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40101200 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3953045344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe400fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- 172.21.15.23:0/3953045344 >> 172.21.15.23:0/3953045344 conn(0x7fbe400fb140 msgr2=0x7fbe400fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- 172.21.15.23:0/3953045344 shutdown_connections 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- 172.21.15.23:0/3953045344 wait complete. 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 Processor -- start 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.810+0000 7fbe452c2640 1 -- start start 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe40197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:04.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 0x7fbe4019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4019f010 con 0x7fbe40100d80 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4019f180 con 0x7fbe400ffb90 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4019f340 con 0x7fbe40101740 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe40197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60216/0 (socket says 172.21.15.23:60216) 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 -- 172.21.15.23:0/3682957662 learned_addr learned my addr 172.21.15.23:0/3682957662 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3f7fe640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 0x7fbe4019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 -- 172.21.15.23:0/3682957662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 msgr2=0x7fbe4019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 0x7fbe4019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 -- 172.21.15.23:0/3682957662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 msgr2=0x7fbe40197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe40197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:04.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 -- 172.21.15.23:0/3682957662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe34009560 con 0x7fbe40100d80 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3effd640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe40197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe3e7fc640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006010 tx=0x7fbe34005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe340043d0 con 0x7fbe40100d80 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe340052d0 con 0x7fbe40100d80 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4019f5c0 con 0x7fbe40100d80 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34003cc0 con 0x7fbe40100d80 2024-03-16T20:36:04.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.811+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4019fad0 con 0x7fbe40100d80 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.813+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe40106e50 con 0x7fbe40100d80 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.814+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbe34005440 con 0x7fbe40100d80 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.815+0000 7fbe27fff640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 0x7fbe280a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.815+0000 7fbe3effd640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 0x7fbe280a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.815+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fbe340eff10 con 0x7fbe40100d80 2024-03-16T20:36:04.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.815+0000 7fbe3effd640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 0x7fbe280a3f90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe40100be0 tx=0x7fbe38010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:04.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:04.818+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbe340b8de0 con 0x7fbe40100d80 2024-03-16T20:36:05.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.040+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fbe4002cf60 con 0x7fbe40100d80 2024-03-16T20:36:05.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.055+0000 7fbe27fff640 1 -- 172.21.15.23:0/3682957662 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe340bc4a0 con 0x7fbe40100d80 2024-03-16T20:36:05.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.057+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 msgr2=0x7fbe280a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.057+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 0x7fbe280a3f90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe40100be0 tx=0x7fbe38010040 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 msgr2=0x7fbe40197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006010 tx=0x7fbe34005e80 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 shutdown_connections 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbe280a1ad0 0x7fbe280a3f90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe40101740 0x7fbe4019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40100d80 0x7fbe40197a60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 --2- 172.21.15.23:0/3682957662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe400ffb90 0x7fbe40197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 >> 172.21.15.23:0/3682957662 conn(0x7fbe400fb140 msgr2=0x7fbe40104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 shutdown_connections 2024-03-16T20:36:05.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.058+0000 7fbe452c2640 1 -- 172.21.15.23:0/3682957662 wait complete. 2024-03-16T20:36:05.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-16T20:36:05.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2534260842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68101fe0 msgr2=0x7fbb68108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2534260842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68101fe0 0x7fbb68108870 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009ca0 tx=0x7fbb6002c0f0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2534260842 shutdown_connections 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2534260842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68101fe0 0x7fbb68108870 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2534260842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101620 0x7fbb68101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2534260842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68100430 0x7fbb68100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2534260842 >> 172.21.15.23:0/2534260842 conn(0x7fbb680fb9a0 msgr2=0x7fbb680fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2534260842 shutdown_connections 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.174+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2534260842 wait complete. 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 Processor -- start 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 -- start start 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68101620 0x7fbb681979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:05.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101fe0 0x7fbb6819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6819ef70 con 0x7fbb68100430 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6819f0e0 con 0x7fbb68101fe0 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6e8d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6819f2a0 con 0x7fbb68101620 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb67fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb67fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60232/0 (socket says 172.21.15.23:60232) 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb67fff640 1 -- 172.21.15.23:0/2052953655 learned_addr learned my addr 172.21.15.23:0/2052953655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb677fe640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68101620 0x7fbb681979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.175+0000 7fbb6ce71640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101fe0 0x7fbb6819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 -- 172.21.15.23:0/2052953655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68101620 msgr2=0x7fbb681979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68101620 0x7fbb681979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 -- 172.21.15.23:0/2052953655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101fe0 msgr2=0x7fbb6819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101fe0 0x7fbb6819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 -- 172.21.15.23:0/2052953655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb60009900 con 0x7fbb68100430 2024-03-16T20:36:05.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb67fff640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800e9c0 tx=0x7fbb5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb580096e0 con 0x7fbb68100430 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb5800cd80 con 0x7fbb68100430 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58010430 con 0x7fbb68100430 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6819f580 con 0x7fbb68100430 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.176+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6819fa40 con 0x7fbb68100430 2024-03-16T20:36:05.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.177+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6810be40 con 0x7fbb68100430 2024-03-16T20:36:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.178+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbb58003680 con 0x7fbb68100430 2024-03-16T20:36:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.178+0000 7fbb657fa640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 0x7fbb500a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:36:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.179+0000 7fbb677fe640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 0x7fbb500a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:36:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.179+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fbb58014070 con 0x7fbb68100430 2024-03-16T20:36:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.179+0000 7fbb677fe640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 0x7fbb500a3f90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb68198990 tx=0x7fbb5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:36:05.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.183+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbb580969a0 con 0x7fbb68100430 2024-03-16T20:36:05.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T20:36:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.392+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 --> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb680fda40 con 0x7fbb500a1ad0 2024-03-16T20:36:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.412+0000 7fbb657fa640 1 -- 172.21.15.23:0/2052953655 <== mgr.14152 v2:172.21.15.23:6800/3040096369 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fbb680fda40 con 0x7fbb500a1ad0 2024-03-16T20:36:05.416 INFO:teuthology.orchestra.run.smithi023.stdout:{} 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 msgr2=0x7fbb500a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 0x7fbb500a3f90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb68198990 tx=0x7fbb5c005f90 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 msgr2=0x7fbb68197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800e9c0 tx=0x7fbb5800ee80 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 shutdown_connections 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbb500a1ad0 0x7fbb500a3f90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb68101fe0 0x7fbb6819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb68101620 0x7fbb681979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 --2- 172.21.15.23:0/2052953655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68100430 0x7fbb68197480 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 >> 172.21.15.23:0/2052953655 conn(0x7fbb680fb9a0 msgr2=0x7fbb680fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 shutdown_connections 2024-03-16T20:36:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:36:05.414+0000 7fbb6e8d4640 1 -- 172.21.15.23:0/2052953655 wait complete. 2024-03-16T20:36:05.429 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-16T20:36:05.439 INFO:teuthology.run_tasks:Running task workunit... 2024-03-16T20:36:05.487 INFO:tasks.workunit:Pulling workunits from ref aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:36:05.487 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-16T20:36:05.487 INFO:tasks.workunit:timeout=3h 2024-03-16T20:36:05.487 INFO:tasks.workunit:cleanup=True 2024-03-16T20:36:05.488 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:05.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:05 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T20:36:05.489 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:05 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-16 20:34:12.088162282 +0000 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-16 20:34:12.088162282 +0000 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-16 20:36:04.362531712 +0000 2024-03-16T20:36:05.503 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-16 20:34:12.088162282 +0000 2024-03-16T20:36:05.503 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-16T20:36:05.504 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-16T20:36:05.577 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b 2024-03-16T20:36:05.633 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-16T20:36:05.840 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:36:05.396+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:36:05.840 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:36:05.396+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:36:05.840 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:36:05.396+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:36:05.840 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:36:05.396+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:36:05.840 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:36:05 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:36:05.396+0000 7f3d023bb700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-16T20:36:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:06 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T20:36:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:06 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:36:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:06 smithi023 ceph-mon[31879]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:36:06.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:06 smithi023 ceph-mon[31879]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T20:36:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:06 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T20:36:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:06 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:36:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:06 smithi031 ceph-mon[36281]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:36:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:06 smithi031 ceph-mon[36281]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T20:36:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:06 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3682957662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-16T20:36:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:06 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:36:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:06 smithi189 ceph-mon[36655]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T20:36:06.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:06 smithi189 ceph-mon[36655]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-16T20:36:08.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:08 smithi023 ceph-mon[31879]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T20:36:08.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:08 smithi023 ceph-mon[31879]: pgmap v361: 129 pgs: 129 active+clean; 656 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T20:36:08.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:08 smithi031 ceph-mon[36281]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T20:36:08.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:08 smithi031 ceph-mon[36281]: pgmap v361: 129 pgs: 129 active+clean; 656 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T20:36:08.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:08 smithi189 ceph-mon[36655]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-16T20:36:08.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:08 smithi189 ceph-mon[36655]: pgmap v361: 129 pgs: 129 active+clean; 656 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-16T20:36:10.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:10 smithi023 ceph-mon[31879]: pgmap v362: 129 pgs: 129 active+clean; 657 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-16T20:36:10.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:10 smithi031 ceph-mon[36281]: pgmap v362: 129 pgs: 129 active+clean; 657 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-16T20:36:10.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:10 smithi189 ceph-mon[36655]: pgmap v362: 129 pgs: 129 active+clean; 657 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-16T20:36:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:12 smithi023 ceph-mon[31879]: pgmap v363: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-16T20:36:12.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:12 smithi031 ceph-mon[36281]: pgmap v363: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-16T20:36:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:12 smithi189 ceph-mon[36655]: pgmap v363: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-16T20:36:14.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:14 smithi023 ceph-mon[31879]: pgmap v364: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T20:36:14.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:14 smithi031 ceph-mon[36281]: pgmap v364: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T20:36:14.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:14 smithi189 ceph-mon[36655]: pgmap v364: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-03-16T20:36:16.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:16 smithi023 ceph-mon[31879]: pgmap v365: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-03-16T20:36:16.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:16 smithi031 ceph-mon[36281]: pgmap v365: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-03-16T20:36:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:16 smithi189 ceph-mon[36655]: pgmap v365: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.3 KiB/s wr, 12 op/s 2024-03-16T20:36:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:18 smithi023 ceph-mon[31879]: pgmap v366: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-16T20:36:18.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:18 smithi031 ceph-mon[36281]: pgmap v366: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-16T20:36:18.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:18 smithi189 ceph-mon[36655]: pgmap v366: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-16T20:36:20.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:20 smithi023 ceph-mon[31879]: pgmap v367: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:20 smithi031 ceph-mon[36281]: pgmap v367: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:20 smithi189 ceph-mon[36655]: pgmap v367: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:21 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:36:21.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:21 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:36:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:21 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:36:22.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:22 smithi023 ceph-mon[31879]: pgmap v368: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-16T20:36:22.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:22 smithi031 ceph-mon[36281]: pgmap v368: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-16T20:36:22.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:22 smithi189 ceph-mon[36655]: pgmap v368: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:23 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:23 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:23.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:23 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:24.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:36:24.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:36:24.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:24.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:24 smithi023 ceph-mon[31879]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:24.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:36:24.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:36:24.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:24.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:24 smithi031 ceph-mon[36281]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:24.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:36:24.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:36:24.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:36:24.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:24 smithi189 ceph-mon[36655]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:26.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:26 smithi023 ceph-mon[31879]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:26.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:26 smithi031 ceph-mon[36281]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:26.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:26 smithi189 ceph-mon[36655]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:28.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:28 smithi023 ceph-mon[31879]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:28.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:28 smithi031 ceph-mon[36281]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:28 smithi189 ceph-mon[36655]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:30 smithi023 ceph-mon[31879]: pgmap v372: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:30.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:30 smithi031 ceph-mon[36281]: pgmap v372: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:30.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:30 smithi189 ceph-mon[36655]: pgmap v372: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:32.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:32 smithi023 ceph-mon[31879]: pgmap v373: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:32.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:32 smithi031 ceph-mon[36281]: pgmap v373: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:32.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:32 smithi189 ceph-mon[36655]: pgmap v373: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:36:34.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:34 smithi023 ceph-mon[31879]: pgmap v374: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:34.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:34 smithi031 ceph-mon[36281]: pgmap v374: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:34.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:34 smithi189 ceph-mon[36655]: pgmap v374: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:36.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:36 smithi023 ceph-mon[31879]: pgmap v375: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:36.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:36 smithi031 ceph-mon[36281]: pgmap v375: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:36 smithi189 ceph-mon[36655]: pgmap v375: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:38.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:38 smithi023 ceph-mon[31879]: pgmap v376: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:38.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:38 smithi031 ceph-mon[36281]: pgmap v376: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:38.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:38 smithi189 ceph-mon[36655]: pgmap v376: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:40.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:40 smithi023 ceph-mon[31879]: pgmap v377: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:40.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:40 smithi031 ceph-mon[36281]: pgmap v377: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:40.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:40 smithi189 ceph-mon[36655]: pgmap v377: 129 pgs: 129 active+clean; 659 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:36:42.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:42 smithi023 ceph-mon[31879]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-16T20:36:42.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:42 smithi031 ceph-mon[36281]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-16T20:36:42.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:42 smithi189 ceph-mon[36655]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-16T20:36:44.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:44 smithi023 ceph-mon[31879]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T20:36:44.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:44 smithi031 ceph-mon[36281]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T20:36:44.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:44 smithi189 ceph-mon[36655]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-16T20:36:46.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:46 smithi023 ceph-mon[31879]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:46.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:46 smithi031 ceph-mon[36281]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:46.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:46 smithi189 ceph-mon[36655]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:48.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:48 smithi023 ceph-mon[31879]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:48.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:48 smithi031 ceph-mon[36281]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:48.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:48 smithi189 ceph-mon[36655]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-16T20:36:50.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:50 smithi023 ceph-mon[31879]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T20:36:50.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:50 smithi031 ceph-mon[36281]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T20:36:50.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:50 smithi189 ceph-mon[36655]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-16T20:36:52.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:52 smithi023 ceph-mon[31879]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-16T20:36:52.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:52 smithi031 ceph-mon[36281]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-16T20:36:52.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:52 smithi189 ceph-mon[36655]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-16T20:36:54.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:54 smithi023 ceph-mon[31879]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:54 smithi031 ceph-mon[36281]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:54 smithi189 ceph-mon[36655]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:36:56.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:56 smithi023 ceph-mon[31879]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:56 smithi031 ceph-mon[36281]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:56 smithi189 ceph-mon[36655]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:36:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:36:58 smithi023 ceph-mon[31879]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:58.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:36:58 smithi031 ceph-mon[36281]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:36:58.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:36:58 smithi189 ceph-mon[36655]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:00 smithi023 ceph-mon[31879]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:00 smithi031 ceph-mon[36281]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:00.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:00 smithi189 ceph-mon[36655]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:02 smithi023 ceph-mon[31879]: pgmap v388: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:02 smithi031 ceph-mon[36281]: pgmap v388: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:02.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:02 smithi189 ceph-mon[36655]: pgmap v388: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:04 smithi023 ceph-mon[31879]: pgmap v389: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:04.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:04 smithi031 ceph-mon[36281]: pgmap v389: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:04.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:04 smithi189 ceph-mon[36655]: pgmap v389: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:06 smithi023 ceph-mon[31879]: pgmap v390: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:06.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:06 smithi031 ceph-mon[36281]: pgmap v390: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:06 smithi189 ceph-mon[36655]: pgmap v390: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:09.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:09 smithi023 ceph-mon[31879]: pgmap v391: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:09.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:09 smithi031 ceph-mon[36281]: pgmap v391: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:09.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:09 smithi189 ceph-mon[36655]: pgmap v391: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:11.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:11 smithi023 ceph-mon[31879]: pgmap v392: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:11.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:11 smithi031 ceph-mon[36281]: pgmap v392: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:11 smithi189 ceph-mon[36655]: pgmap v392: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-16T20:37:13.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:13 smithi023 ceph-mon[31879]: pgmap v393: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:13.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:13 smithi031 ceph-mon[36281]: pgmap v393: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:13.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:13 smithi189 ceph-mon[36655]: pgmap v393: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-16T20:37:15.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:15 smithi023 ceph-mon[31879]: pgmap v394: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:15.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:15 smithi031 ceph-mon[36281]: pgmap v394: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:15.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:15 smithi189 ceph-mon[36655]: pgmap v394: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:37:17.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:17 smithi023 ceph-mon[31879]: pgmap v395: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:17.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:17 smithi031 ceph-mon[36281]: pgmap v395: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:17.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:17 smithi189 ceph-mon[36655]: pgmap v395: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-16T20:37:19.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:19 smithi023 ceph-mon[31879]: pgmap v396: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:19.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:19 smithi031 ceph-mon[36281]: pgmap v396: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:19.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:19 smithi189 ceph-mon[36655]: pgmap v396: 129 pgs: 129 active+clean; 664 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-16T20:37:21.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:21 smithi023 ceph-mon[31879]: pgmap v397: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:37:21.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:21 smithi031 ceph-mon[36281]: pgmap v397: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:37:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:21 smithi189 ceph-mon[36655]: pgmap v397: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:37:23.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:23 smithi023 ceph-mon[31879]: pgmap v398: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T20:37:23.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:23 smithi023 ceph-mon[31879]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T20:37:23.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:23 smithi031 ceph-mon[36281]: pgmap v398: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T20:37:23.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:23 smithi031 ceph-mon[36281]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T20:37:23.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:23 smithi189 ceph-mon[36655]: pgmap v398: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-16T20:37:23.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:23 smithi189 ceph-mon[36655]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-16T20:37:24.391 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 92% (11355/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-16T20:37:24.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:24 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:37:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:24 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:37:24.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:24 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to 'aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b'. 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2024-03-16T20:37:24.875 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-16T20:37:24.876 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at aba3c1d240c mds/quiesce-db: keep the db thread alive until shutdown 2024-03-16T20:37:24.877 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-16T20:37:24.953 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-16T20:37:24.955 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T20:37:24.955 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-16T20:37:25.678 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-16T20:37:25.735 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-16T20:37:25.774 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-16T20:37:25.776 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T20:37:25.776 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-16T20:37:25.815 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-16T20:37:25.818 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:37:25.818 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: pgmap v399: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:37:25.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:25 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.855 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-16T20:37:25.857 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-16T20:37:25.857 DEBUG:teuthology.orchestra.run.smithi023:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-16T20:37:25.917 INFO:tasks.workunit.client.0.smithi023.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-16T20:37:25.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-16T20:37:25.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ trap -- retry EXIT 2024-03-16T20:37:25.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rf ffsb 2024-03-16T20:37:25.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-16T20:37:25.921 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into 'ffsb'... 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: pgmap v399: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:37:25.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:25 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: pgmap v399: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-16T20:37:25.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:37:25.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:25 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:37:27.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ trap - EXIT 2024-03-16T20:37:27.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ffsb 2024-03-16T20:37:27.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ ./configure 2024-03-16T20:37:27.158 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-16T20:37:27.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:27 smithi023 ceph-mon[31879]: pgmap v400: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T20:37:27.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:27 smithi031 ceph-mon[36281]: pgmap v400: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T20:37:27.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:27 smithi189 ceph-mon[36655]: pgmap v400: 129 pgs: 129 active+clean; 665 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-16T20:37:28.167 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether build environment is sane... yes 2024-03-16T20:37:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-16T20:37:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gawk... gawk 2024-03-16T20:37:28.193 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether make sets $(MAKE)... yes 2024-03-16T20:37:28.223 INFO:tasks.workunit.client.0.smithi023.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-16T20:37:28.223 INFO:tasks.workunit.client.0.smithi023.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-16T20:37:28.234 INFO:tasks.workunit.client.0.smithi023.stdout:checking for style of include used by make... GNU 2024-03-16T20:37:28.240 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... gcc 2024-03-16T20:37:28.293 INFO:tasks.workunit.client.0.smithi023.stdout:checking for C compiler default output file name... a.out 2024-03-16T20:37:28.295 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether the C compiler works... yes 2024-03-16T20:37:28.298 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are cross compiling... no 2024-03-16T20:37:28.336 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of executables... 2024-03-16T20:37:28.368 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of object files... o 2024-03-16T20:37:28.398 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... yes 2024-03-16T20:37:28.431 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... yes 2024-03-16T20:37:28.494 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-16T20:37:28.596 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... gcc3 2024-03-16T20:37:28.659 INFO:tasks.workunit.client.0.smithi023.stdout:checking how to run the C preprocessor... gcc -E 2024-03-16T20:37:28.704 INFO:tasks.workunit.client.0.smithi023.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-16T20:37:28.705 INFO:tasks.workunit.client.0.smithi023.stdout:checking for egrep... /usr/bin/grep -E 2024-03-16T20:37:28.713 INFO:tasks.workunit.client.0.smithi023.stdout:checking for AIX... no 2024-03-16T20:37:28.740 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... (cached) gcc 2024-03-16T20:37:28.747 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-16T20:37:28.748 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-16T20:37:28.749 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-16T20:37:28.749 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-16T20:37:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lm... yes 2024-03-16T20:37:28.849 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lpthread... yes 2024-03-16T20:37:29.010 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ANSI C header files... yes 2024-03-16T20:37:29.050 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-16T20:37:29.095 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/types.h... yes 2024-03-16T20:37:29.142 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/stat.h... yes 2024-03-16T20:37:29.191 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdlib.h... yes 2024-03-16T20:37:29.238 INFO:tasks.workunit.client.0.smithi023.stdout:checking for string.h... yes 2024-03-16T20:37:29.300 INFO:tasks.workunit.client.0.smithi023.stdout:checking for memory.h... yes 2024-03-16T20:37:29.345 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strings.h... yes 2024-03-16T20:37:29.396 INFO:tasks.workunit.client.0.smithi023.stdout:checking for inttypes.h... yes 2024-03-16T20:37:29.441 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... yes 2024-03-16T20:37:29.488 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... yes 2024-03-16T20:37:29.544 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h usability... yes 2024-03-16T20:37:29.565 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h presence... yes 2024-03-16T20:37:29.565 INFO:tasks.workunit.client.0.smithi023.stdout:checking for pthread.h... yes 2024-03-16T20:37:29.618 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h usability... yes 2024-03-16T20:37:29.639 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h presence... yes 2024-03-16T20:37:29.639 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fcntl.h... yes 2024-03-16T20:37:29.698 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h usability... yes 2024-03-16T20:37:29.720 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h presence... yes 2024-03-16T20:37:29.720 INFO:tasks.workunit.client.0.smithi023.stdout:checking for limits.h... yes 2024-03-16T20:37:29.738 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... (cached) yes 2024-03-16T20:37:29.788 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h usability... yes 2024-03-16T20:37:29.806 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h presence... yes 2024-03-16T20:37:29.807 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/time.h... yes 2024-03-16T20:37:29.825 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... (cached) yes 2024-03-16T20:37:29.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:29 smithi023 ceph-mon[31879]: pgmap v401: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 96 KiB/s wr, 16 op/s 2024-03-16T20:37:29.885 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h usability... yes 2024-03-16T20:37:29.907 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h presence... yes 2024-03-16T20:37:29.907 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/vfs.h... yes 2024-03-16T20:37:29.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:29 smithi031 ceph-mon[36281]: pgmap v401: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 96 KiB/s wr, 16 op/s 2024-03-16T20:37:29.955 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h usability... no 2024-03-16T20:37:29.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:29 smithi189 ceph-mon[36655]: pgmap v401: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 96 KiB/s wr, 16 op/s 2024-03-16T20:37:29.977 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h presence... no 2024-03-16T20:37:29.977 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/limits.h... no 2024-03-16T20:37:30.010 INFO:tasks.workunit.client.0.smithi023.stdout:checking for an ANSI C-conforming const... yes 2024-03-16T20:37:30.046 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-16T20:37:30.046 INFO:tasks.workunit.client.0.smithi023.stdout:checking for function prototypes... yes 2024-03-16T20:37:30.077 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether setvbuf arguments are reversed... no 2024-03-16T20:37:30.152 INFO:tasks.workunit.client.0.smithi023.stdout:checking for system... yes 2024-03-16T20:37:30.220 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gettimeofday... yes 2024-03-16T20:37:30.285 INFO:tasks.workunit.client.0.smithi023.stdout:checking for mkdir... yes 2024-03-16T20:37:30.349 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strerror... yes 2024-03-16T20:37:30.410 INFO:tasks.workunit.client.0.smithi023.stdout:checking for open64... yes 2024-03-16T20:37:30.471 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stat64... yes 2024-03-16T20:37:30.532 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fseeko64... yes 2024-03-16T20:37:30.592 INFO:tasks.workunit.client.0.smithi023.stdout:checking for lrand48_r... yes 2024-03-16T20:37:30.655 INFO:tasks.workunit.client.0.smithi023.stdout:checking for srand48_r... yes 2024-03-16T20:37:30.677 INFO:tasks.workunit.client.0.smithi023.stdout:configure: creating ./config.status 2024-03-16T20:37:30.778 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating Makefile 2024-03-16T20:37:30.791 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating config.h 2024-03-16T20:37:30.807 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing depfiles commands 2024-03-16T20:37:30.852 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing default commands 2024-03-16T20:37:30.905 INFO:tasks.workunit.client.0.smithi023.stderr:+ make 2024-03-16T20:37:30.914 INFO:tasks.workunit.client.0.smithi023.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-16T20:37:30.916 INFO:tasks.workunit.client.0.smithi023.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-16T20:37:30.917 INFO:tasks.workunit.client.0.smithi023.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-16T20:37:30.917 INFO:tasks.workunit.client.0.smithi023.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-16T20:37:30.917 INFO:tasks.workunit.client.0.smithi023.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-16T20:37:30.917 INFO:tasks.workunit.client.0.smithi023.stderr: any GNU archive site. 2024-03-16T20:37:30.919 INFO:tasks.workunit.client.0.smithi023.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-16T20:37:30.920 INFO:tasks.workunit.client.0.smithi023.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-16T20:37:30.921 INFO:tasks.workunit.client.0.smithi023.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-16T20:37:30.921 INFO:tasks.workunit.client.0.smithi023.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-16T20:37:30.921 INFO:tasks.workunit.client.0.smithi023.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-16T20:37:30.921 INFO:tasks.workunit.client.0.smithi023.stderr: Grab them from any GNU archive site. 2024-03-16T20:37:31.018 INFO:tasks.workunit.client.0.smithi023.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-16T20:37:31.487 INFO:tasks.workunit.client.0.smithi023.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-16T20:37:31.487 INFO:tasks.workunit.client.0.smithi023.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-16T20:37:31.487 INFO:tasks.workunit.client.0.smithi023.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-16T20:37:31.487 INFO:tasks.workunit.client.0.smithi023.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-16T20:37:31.685 INFO:tasks.workunit.client.0.smithi023.stdout:/bin/sh ./config.status --recheck 2024-03-16T20:37:31.741 INFO:tasks.workunit.client.0.smithi023.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-16T20:37:31.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:31 smithi023 ceph-mon[31879]: pgmap v402: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 97 KiB/s wr, 16 op/s 2024-03-16T20:37:31.845 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-16T20:37:31.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:31 smithi031 ceph-mon[36281]: pgmap v402: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 97 KiB/s wr, 16 op/s 2024-03-16T20:37:31.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:31 smithi189 ceph-mon[36655]: pgmap v402: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 97 KiB/s wr, 16 op/s 2024-03-16T20:37:32.839 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:32 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:32.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:32 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:32.852 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether build environment is sane... yes 2024-03-16T20:37:32.856 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-16T20:37:32.857 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gawk... gawk 2024-03-16T20:37:32.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:32 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:32.874 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether make sets $(MAKE)... yes 2024-03-16T20:37:32.900 INFO:tasks.workunit.client.0.smithi023.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-16T20:37:32.901 INFO:tasks.workunit.client.0.smithi023.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-16T20:37:32.913 INFO:tasks.workunit.client.0.smithi023.stdout:checking for style of include used by make... GNU 2024-03-16T20:37:32.921 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... gcc 2024-03-16T20:37:33.010 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether the C compiler works... yes 2024-03-16T20:37:33.010 INFO:tasks.workunit.client.0.smithi023.stdout:checking for C compiler default output file name... a.out 2024-03-16T20:37:33.050 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of executables... 2024-03-16T20:37:33.087 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are cross compiling... no 2024-03-16T20:37:33.123 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of object files... o 2024-03-16T20:37:33.146 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... yes 2024-03-16T20:37:33.168 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... yes 2024-03-16T20:37:33.208 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-16T20:37:33.330 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... gcc3 2024-03-16T20:37:33.374 INFO:tasks.workunit.client.0.smithi023.stdout:checking how to run the C preprocessor... gcc -E 2024-03-16T20:37:33.425 INFO:tasks.workunit.client.0.smithi023.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-16T20:37:33.426 INFO:tasks.workunit.client.0.smithi023.stdout:checking for egrep... /usr/bin/grep -E 2024-03-16T20:37:33.562 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ANSI C header files... yes 2024-03-16T20:37:33.595 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/types.h... yes 2024-03-16T20:37:33.639 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/stat.h... yes 2024-03-16T20:37:33.681 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdlib.h... yes 2024-03-16T20:37:33.720 INFO:tasks.workunit.client.0.smithi023.stdout:checking for string.h... yes 2024-03-16T20:37:33.761 INFO:tasks.workunit.client.0.smithi023.stdout:checking for memory.h... yes 2024-03-16T20:37:33.803 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strings.h... yes 2024-03-16T20:37:33.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:33 smithi023 ceph-mon[31879]: pgmap v403: 129 pgs: 129 active+clean; 3.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-16T20:37:33.851 INFO:tasks.workunit.client.0.smithi023.stdout:checking for inttypes.h... yes 2024-03-16T20:37:33.884 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:33 smithi189 ceph-mon[36655]: pgmap v403: 129 pgs: 129 active+clean; 3.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-16T20:37:33.891 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... yes 2024-03-16T20:37:33.933 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... yes 2024-03-16T20:37:33.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:33 smithi031 ceph-mon[36281]: pgmap v403: 129 pgs: 129 active+clean; 3.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 290 KiB/s wr, 26 op/s 2024-03-16T20:37:33.982 INFO:tasks.workunit.client.0.smithi023.stdout:checking minix/config.h usability... no 2024-03-16T20:37:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:checking minix/config.h presence... no 2024-03-16T20:37:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:checking for minix/config.h... no 2024-03-16T20:37:34.051 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-16T20:37:34.051 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... (cached) gcc 2024-03-16T20:37:34.092 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-16T20:37:34.093 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-16T20:37:34.093 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-16T20:37:34.093 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-16T20:37:34.135 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lm... yes 2024-03-16T20:37:34.181 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lpthread... yes 2024-03-16T20:37:34.182 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ANSI C header files... (cached) yes 2024-03-16T20:37:34.214 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-16T20:37:34.281 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h usability... yes 2024-03-16T20:37:34.301 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h presence... yes 2024-03-16T20:37:34.301 INFO:tasks.workunit.client.0.smithi023.stdout:checking for pthread.h... yes 2024-03-16T20:37:34.354 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h usability... yes 2024-03-16T20:37:34.373 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h presence... yes 2024-03-16T20:37:34.373 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fcntl.h... yes 2024-03-16T20:37:34.425 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h usability... yes 2024-03-16T20:37:34.444 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h presence... yes 2024-03-16T20:37:34.444 INFO:tasks.workunit.client.0.smithi023.stdout:checking for limits.h... yes 2024-03-16T20:37:34.459 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... (cached) yes 2024-03-16T20:37:34.496 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h usability... yes 2024-03-16T20:37:34.515 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h presence... yes 2024-03-16T20:37:34.515 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/time.h... yes 2024-03-16T20:37:34.528 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... (cached) yes 2024-03-16T20:37:34.566 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h usability... yes 2024-03-16T20:37:34.584 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h presence... yes 2024-03-16T20:37:34.584 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/vfs.h... yes 2024-03-16T20:37:34.638 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h usability... no 2024-03-16T20:37:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h presence... no 2024-03-16T20:37:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/limits.h... no 2024-03-16T20:37:34.694 INFO:tasks.workunit.client.0.smithi023.stdout:checking for an ANSI C-conforming const... yes 2024-03-16T20:37:34.721 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-16T20:37:34.786 INFO:tasks.workunit.client.0.smithi023.stdout:checking for system... yes 2024-03-16T20:37:34.840 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gettimeofday... yes 2024-03-16T20:37:34.900 INFO:tasks.workunit.client.0.smithi023.stdout:checking for mkdir... yes 2024-03-16T20:37:34.957 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strerror... yes 2024-03-16T20:37:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:checking for open64... yes 2024-03-16T20:37:35.070 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stat64... yes 2024-03-16T20:37:35.130 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fseeko64... yes 2024-03-16T20:37:35.184 INFO:tasks.workunit.client.0.smithi023.stdout:checking for lrand48_r... yes 2024-03-16T20:37:35.241 INFO:tasks.workunit.client.0.smithi023.stdout:checking for srand48_r... yes 2024-03-16T20:37:35.265 INFO:tasks.workunit.client.0.smithi023.stdout:configure: creating ./config.status 2024-03-16T20:37:35.326 INFO:tasks.workunit.client.0.smithi023.stdout: /bin/sh ./config.status 2024-03-16T20:37:35.370 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating Makefile 2024-03-16T20:37:35.381 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating config.h 2024-03-16T20:37:35.400 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing depfiles commands 2024-03-16T20:37:35.410 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing default commands 2024-03-16T20:37:35.445 INFO:tasks.workunit.client.0.smithi023.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-16T20:37:35.607 INFO:tasks.workunit.client.0.smithi023.stdout:rm -f stamp-h1 2024-03-16T20:37:35.610 INFO:tasks.workunit.client.0.smithi023.stdout:touch config.h.in 2024-03-16T20:37:35.611 INFO:tasks.workunit.client.0.smithi023.stdout:cd . && /bin/sh ./config.status config.h 2024-03-16T20:37:35.653 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating config.h 2024-03-16T20:37:35.685 INFO:tasks.workunit.client.0.smithi023.stdout:make all-am 2024-03-16T20:37:35.692 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-16T20:37:35.692 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-16T20:37:35.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:35 smithi189 ceph-mon[36655]: pgmap v404: 129 pgs: 129 active+clean; 4.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 307 KiB/s wr, 27 op/s 2024-03-16T20:37:35.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:35 smithi023 ceph-mon[31879]: pgmap v404: 129 pgs: 129 active+clean; 4.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 307 KiB/s wr, 27 op/s 2024-03-16T20:37:35.867 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-16T20:37:35.871 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-16T20:37:35.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:35 smithi031 ceph-mon[36281]: pgmap v404: 129 pgs: 129 active+clean; 4.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 307 KiB/s wr, 27 op/s 2024-03-16T20:37:35.952 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-16T20:37:35.956 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-16T20:37:36.036 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-16T20:37:36.039 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-16T20:37:36.125 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-16T20:37:36.129 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-16T20:37:36.247 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-16T20:37:36.251 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-16T20:37:36.300 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-16T20:37:36.302 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-16T20:37:36.342 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-16T20:37:36.345 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-16T20:37:36.388 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-16T20:37:36.391 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-16T20:37:36.596 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-16T20:37:36.599 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-16T20:37:36.722 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-16T20:37:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-16T20:37:36.867 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-16T20:37:36.870 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-16T20:37:36.937 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-16T20:37:36.938 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-16T20:37:37.006 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-16T20:37:37.009 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-16T20:37:37.109 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-16T20:37:37.112 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr:In file included from parser.h:22, 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr: from parser.c:25: 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘init_filesys’: 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.140 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-16T20:37:37.141 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.142 INFO:tasks.workunit.client.0.smithi023.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.143 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.144 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.145 INFO:tasks.workunit.client.0.smithi023.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.146 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.147 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.148 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.149 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: | ^ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: 36 | container_of(ptr, type, member) 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~ 2024-03-16T20:37:37.150 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-16T20:37:37.151 INFO:tasks.workunit.client.0.smithi023.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-16T20:37:37.151 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~ 2024-03-16T20:37:37.151 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-16T20:37:37.151 INFO:tasks.workunit.client.0.smithi023.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-16T20:37:37.151 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘get_optstr’: 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:200:14: note: source object declared here 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr: 200 | char temp[BUFSIZE]; 2024-03-16T20:37:37.188 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~ 2024-03-16T20:37:37.194 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘size64_convert’: 2024-03-16T20:37:37.194 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-16T20:37:37.194 INFO:tasks.workunit.client.0.smithi023.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-16T20:37:37.194 INFO:tasks.workunit.client.0.smithi023.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-16T20:37:37.486 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-16T20:37:37.489 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-16T20:37:37.544 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-16T20:37:37.547 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-16T20:37:37.637 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-16T20:37:37.639 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-16T20:37:37.668 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-16T20:37:37.671 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-16T20:37:37.702 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-16T20:37:37.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .. 2024-03-16T20:37:37.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir tmp 2024-03-16T20:37:37.705 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd tmp 2024-03-16T20:37:37.705 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in $mydir/*.ffsb 2024-03-16T20:37:37.705 INFO:tasks.workunit.client.0.smithi023.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-16T20:37:37.705 INFO:tasks.workunit.client.0.smithi023.stdout:FFSB version 6.0-RC2 started 2024-03-16T20:37:37.706 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout:benchmark time = 300 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout:ThreadGroup 0 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout:================ 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout: num_threads = 32 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.708 INFO:tasks.workunit.client.0.smithi023.stdout: read_random = off 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: read_size = 0 (0B) 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: read_blocksize = 0 (0B) 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: read_skip = off 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: read_skipsize = 0 (0B) 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: write_random = on 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: write_size = 5242880 (5MB) 2024-03-16T20:37:37.709 INFO:tasks.workunit.client.0.smithi023.stdout: fsync_file = 0 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: write_blocksize = 4096 (4KB) 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: wait time = 0 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: op weights 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: read = 0 (0.00%) 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: readall = 0 (0.00%) 2024-03-16T20:37:37.710 INFO:tasks.workunit.client.0.smithi023.stdout: write = 1 (100.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: create = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: append = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: delete = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: metaop = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: createdir = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: stat = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: writeall = 0 (0.00%) 2024-03-16T20:37:37.711 INFO:tasks.workunit.client.0.smithi023.stdout: writeall_fsync = 0 (0.00%) 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: open_close = 0 (0.00%) 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: write_fsync = 0 (0.00%) 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: create_fsync = 0 (0.00%) 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: append_fsync = 0 (0.00%) 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout:FileSystem . 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout:========== 2024-03-16T20:37:37.712 INFO:tasks.workunit.client.0.smithi023.stdout: num_dirs = 0 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: starting files = 128 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: min file size = 104857600 (100MB) 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: max file size = 104857600 (100MB) 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: directio = off 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: alignedio = on 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: bufferedio = off 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.713 INFO:tasks.workunit.client.0.smithi023.stdout: aging is off 2024-03-16T20:37:37.714 INFO:tasks.workunit.client.0.smithi023.stdout: current utilization = 0.03% 2024-03-16T20:37:37.714 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:37.714 INFO:tasks.workunit.client.0.smithi023.stdout:checking existing fs: . 2024-03-16T20:37:37.714 INFO:tasks.workunit.client.0.smithi023.stdout:recreating new fileset 2024-03-16T20:37:37.714 INFO:tasks.workunit.client.0.smithi023.stderr:opendir: No such file or directory 2024-03-16T20:37:37.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:37 smithi023 ceph-mon[31879]: pgmap v405: 129 pgs: 129 active+clean; 4.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 428 KiB/s wr, 54 op/s 2024-03-16T20:37:37.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:37 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:37.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:37 smithi031 ceph-mon[36281]: pgmap v405: 129 pgs: 129 active+clean; 4.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 428 KiB/s wr, 54 op/s 2024-03-16T20:37:37.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:37 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:37.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:37 smithi189 ceph-mon[36655]: pgmap v405: 129 pgs: 129 active+clean; 4.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 428 KiB/s wr, 54 op/s 2024-03-16T20:37:37.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:37 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:37:39.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:39 smithi023 ceph-mon[31879]: pgmap v406: 129 pgs: 129 active+clean; 5.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 512 KiB/s wr, 66 op/s 2024-03-16T20:37:39.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:39 smithi031 ceph-mon[36281]: pgmap v406: 129 pgs: 129 active+clean; 5.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 512 KiB/s wr, 66 op/s 2024-03-16T20:37:39.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:39 smithi189 ceph-mon[36655]: pgmap v406: 129 pgs: 129 active+clean; 5.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 512 KiB/s wr, 66 op/s 2024-03-16T20:37:41.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:41 smithi023 ceph-mon[31879]: pgmap v407: 129 pgs: 129 active+clean; 100 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.3 MiB/s wr, 93 op/s 2024-03-16T20:37:41.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:41 smithi031 ceph-mon[36281]: pgmap v407: 129 pgs: 129 active+clean; 100 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.3 MiB/s wr, 93 op/s 2024-03-16T20:37:41.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:41 smithi189 ceph-mon[36655]: pgmap v407: 129 pgs: 129 active+clean; 100 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.3 MiB/s wr, 93 op/s 2024-03-16T20:37:43.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:43 smithi023 ceph-mon[31879]: pgmap v408: 129 pgs: 129 active+clean; 696 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 58 MiB/s wr, 121 op/s 2024-03-16T20:37:43.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:43 smithi031 ceph-mon[36281]: pgmap v408: 129 pgs: 129 active+clean; 696 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 58 MiB/s wr, 121 op/s 2024-03-16T20:37:43.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:43 smithi189 ceph-mon[36655]: pgmap v408: 129 pgs: 129 active+clean; 696 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 58 MiB/s wr, 121 op/s 2024-03-16T20:37:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:45 smithi189 ceph-mon[36655]: pgmap v409: 129 pgs: 129 active+clean; 1.3 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 109 MiB/s wr, 127 op/s 2024-03-16T20:37:45.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:45 smithi023 ceph-mon[31879]: pgmap v409: 129 pgs: 129 active+clean; 1.3 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 109 MiB/s wr, 127 op/s 2024-03-16T20:37:45.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:45 smithi031 ceph-mon[36281]: pgmap v409: 129 pgs: 129 active+clean; 1.3 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 109 MiB/s wr, 127 op/s 2024-03-16T20:37:47.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:47 smithi023 ceph-mon[31879]: pgmap v410: 129 pgs: 129 active+clean; 4.0 GiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 341 MiB/s wr, 231 op/s 2024-03-16T20:37:47.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:47 smithi031 ceph-mon[36281]: pgmap v410: 129 pgs: 129 active+clean; 4.0 GiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 341 MiB/s wr, 231 op/s 2024-03-16T20:37:47.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:47 smithi189 ceph-mon[36655]: pgmap v410: 129 pgs: 129 active+clean; 4.0 GiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 341 MiB/s wr, 231 op/s 2024-03-16T20:37:48.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:48 smithi023 ceph-mon[31879]: pgmap v411: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 461 MiB/s wr, 237 op/s 2024-03-16T20:37:48.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:48 smithi031 ceph-mon[36281]: pgmap v411: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 461 MiB/s wr, 237 op/s 2024-03-16T20:37:48.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:48 smithi189 ceph-mon[36655]: pgmap v411: 129 pgs: 129 active+clean; 5.4 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 461 MiB/s wr, 237 op/s 2024-03-16T20:37:49.911 INFO:tasks.workunit.client.0.smithi023.stdout:fs setup took 12 secs 2024-03-16T20:37:50.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:50 smithi023 ceph-mon[31879]: pgmap v412: 129 pgs: 129 active+clean; 7.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 642 MiB/s wr, 306 op/s 2024-03-16T20:37:50.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:50 smithi031 ceph-mon[36281]: pgmap v412: 129 pgs: 129 active+clean; 7.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 642 MiB/s wr, 306 op/s 2024-03-16T20:37:50.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:50 smithi189 ceph-mon[36655]: pgmap v412: 129 pgs: 129 active+clean; 7.5 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 642 MiB/s wr, 306 op/s 2024-03-16T20:37:52.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:52 smithi023 ceph-mon[31879]: pgmap v413: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 819 MiB/s wr, 321 op/s 2024-03-16T20:37:52.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:52 smithi031 ceph-mon[36281]: pgmap v413: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 819 MiB/s wr, 321 op/s 2024-03-16T20:37:52.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:52 smithi189 ceph-mon[36655]: pgmap v413: 129 pgs: 129 active+clean; 9.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 819 MiB/s wr, 321 op/s 2024-03-16T20:37:53.117 INFO:tasks.workunit.client.0.smithi023.stdout:Syncing()...3 sec 2024-03-16T20:37:53.117 INFO:tasks.workunit.client.0.smithi023.stdout:Starting Actual Benchmark At: Sat Mar 16 20:37:53 2024 2024-03-16T20:37:53.117 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:37:54.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:54 smithi023 ceph-mon[31879]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 830 MiB/s wr, 353 op/s 2024-03-16T20:37:54.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:54 smithi031 ceph-mon[36281]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 830 MiB/s wr, 353 op/s 2024-03-16T20:37:54.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:54 smithi189 ceph-mon[36655]: pgmap v414: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 830 MiB/s wr, 353 op/s 2024-03-16T20:37:56.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:56 smithi031 ceph-mon[36281]: pgmap v415: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 927 MiB/s wr, 1.39k op/s 2024-03-16T20:37:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:56 smithi189 ceph-mon[36655]: pgmap v415: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 927 MiB/s wr, 1.39k op/s 2024-03-16T20:37:56.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:56 smithi023 ceph-mon[31879]: pgmap v415: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 927 MiB/s wr, 1.39k op/s 2024-03-16T20:37:58.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:37:58 smithi023 ceph-mon[31879]: pgmap v416: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 739 MiB/s wr, 2.50k op/s 2024-03-16T20:37:58.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:37:58 smithi031 ceph-mon[36281]: pgmap v416: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 739 MiB/s wr, 2.50k op/s 2024-03-16T20:37:58.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:37:58 smithi189 ceph-mon[36655]: pgmap v416: 129 pgs: 129 active+clean; 13 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 739 MiB/s wr, 2.50k op/s 2024-03-16T20:38:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:00 smithi023 ceph-mon[31879]: pgmap v417: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 625 MiB/s wr, 3.38k op/s 2024-03-16T20:38:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:00 smithi031 ceph-mon[36281]: pgmap v417: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 625 MiB/s wr, 3.38k op/s 2024-03-16T20:38:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:00 smithi189 ceph-mon[36655]: pgmap v417: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 625 MiB/s wr, 3.38k op/s 2024-03-16T20:38:02.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:02 smithi023 ceph-mon[31879]: pgmap v418: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 480 MiB/s wr, 8.64k op/s 2024-03-16T20:38:02.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:02 smithi031 ceph-mon[36281]: pgmap v418: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 480 MiB/s wr, 8.64k op/s 2024-03-16T20:38:02.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:02 smithi189 ceph-mon[36655]: pgmap v418: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 480 MiB/s wr, 8.64k op/s 2024-03-16T20:38:04.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:04 smithi023 ceph-mon[31879]: pgmap v419: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 MiB/s wr, 9.53k op/s 2024-03-16T20:38:04.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:04 smithi031 ceph-mon[36281]: pgmap v419: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 MiB/s wr, 9.53k op/s 2024-03-16T20:38:04.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:04 smithi189 ceph-mon[36655]: pgmap v419: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 MiB/s wr, 9.53k op/s 2024-03-16T20:38:07.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:07 smithi023 ceph-mon[31879]: pgmap v420: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 274 MiB/s wr, 13.68k op/s 2024-03-16T20:38:07.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:07 smithi031 ceph-mon[36281]: pgmap v420: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 274 MiB/s wr, 13.68k op/s 2024-03-16T20:38:07.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:07 smithi189 ceph-mon[36655]: pgmap v420: 129 pgs: 129 active+clean; 13 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 274 MiB/s wr, 13.68k op/s 2024-03-16T20:38:09.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:09 smithi023 ceph-mon[31879]: pgmap v421: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 140 MiB/s wr, 14.39k op/s 2024-03-16T20:38:09.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:09 smithi031 ceph-mon[36281]: pgmap v421: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 140 MiB/s wr, 14.39k op/s 2024-03-16T20:38:09.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:09 smithi189 ceph-mon[36655]: pgmap v421: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 140 MiB/s wr, 14.39k op/s 2024-03-16T20:38:11.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:11 smithi023 ceph-mon[31879]: pgmap v422: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 104 MiB/s wr, 14.15k op/s 2024-03-16T20:38:11.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:11 smithi031 ceph-mon[36281]: pgmap v422: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 104 MiB/s wr, 14.15k op/s 2024-03-16T20:38:11.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:11 smithi189 ceph-mon[36655]: pgmap v422: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 104 MiB/s wr, 14.15k op/s 2024-03-16T20:38:13.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:13 smithi023 ceph-mon[31879]: pgmap v423: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 139 MiB/s wr, 18.51k op/s 2024-03-16T20:38:13.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:13 smithi031 ceph-mon[36281]: pgmap v423: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 139 MiB/s wr, 18.51k op/s 2024-03-16T20:38:13.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:13 smithi189 ceph-mon[36655]: pgmap v423: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 139 MiB/s wr, 18.51k op/s 2024-03-16T20:38:15.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:15 smithi031 ceph-mon[36281]: pgmap v424: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 110 MiB/s wr, 14k op/s 2024-03-16T20:38:15.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:15 smithi189 ceph-mon[36655]: pgmap v424: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 110 MiB/s wr, 14k op/s 2024-03-16T20:38:15.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:15 smithi023 ceph-mon[31879]: pgmap v424: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 110 MiB/s wr, 14k op/s 2024-03-16T20:38:17.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:17 smithi031 ceph-mon[36281]: pgmap v425: 129 pgs: 129 active+clean; 13 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 140 MiB/s wr, 17.45k op/s 2024-03-16T20:38:17.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:17 smithi189 ceph-mon[36655]: pgmap v425: 129 pgs: 129 active+clean; 13 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 140 MiB/s wr, 17.45k op/s 2024-03-16T20:38:17.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:17 smithi023 ceph-mon[31879]: pgmap v425: 129 pgs: 129 active+clean; 13 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 140 MiB/s wr, 17.45k op/s 2024-03-16T20:38:19.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:19 smithi031 ceph-mon[36281]: pgmap v426: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 124 MiB/s wr, 15.13k op/s 2024-03-16T20:38:19.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:19 smithi189 ceph-mon[36655]: pgmap v426: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 124 MiB/s wr, 15.13k op/s 2024-03-16T20:38:19.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:19 smithi023 ceph-mon[31879]: pgmap v426: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 124 MiB/s wr, 15.13k op/s 2024-03-16T20:38:21.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:21 smithi031 ceph-mon[36281]: pgmap v427: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 116 MiB/s wr, 14.11k op/s 2024-03-16T20:38:21.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:21 smithi189 ceph-mon[36655]: pgmap v427: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 116 MiB/s wr, 14.11k op/s 2024-03-16T20:38:21.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:21 smithi023 ceph-mon[31879]: pgmap v427: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 116 MiB/s wr, 14.11k op/s 2024-03-16T20:38:23.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:23 smithi031 ceph-mon[36281]: pgmap v428: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 147 MiB/s wr, 17.47k op/s 2024-03-16T20:38:23.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:23 smithi189 ceph-mon[36655]: pgmap v428: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 147 MiB/s wr, 17.47k op/s 2024-03-16T20:38:23.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:23 smithi023 ceph-mon[31879]: pgmap v428: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 147 MiB/s wr, 17.47k op/s 2024-03-16T20:38:25.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:25 smithi031 ceph-mon[36281]: pgmap v429: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 117 MiB/s wr, 13.44k op/s 2024-03-16T20:38:25.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:25 smithi189 ceph-mon[36655]: pgmap v429: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 117 MiB/s wr, 13.44k op/s 2024-03-16T20:38:25.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:25 smithi023 ceph-mon[31879]: pgmap v429: 129 pgs: 129 active+clean; 13 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 117 MiB/s wr, 13.44k op/s 2024-03-16T20:38:26.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:26 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:38:26.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:26 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:38:26.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:26 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:38:27.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:27 smithi031 ceph-mon[36281]: pgmap v430: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 131 MiB/s wr, 15.16k op/s 2024-03-16T20:38:27.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:38:27.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:38:27.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:38:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:27 smithi189 ceph-mon[36655]: pgmap v430: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 131 MiB/s wr, 15.16k op/s 2024-03-16T20:38:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:38:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:38:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:38:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:27 smithi023 ceph-mon[31879]: pgmap v430: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 131 MiB/s wr, 15.16k op/s 2024-03-16T20:38:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:38:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:38:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:38:29.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:29 smithi031 ceph-mon[36281]: pgmap v431: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 126 MiB/s wr, 14.52k op/s 2024-03-16T20:38:29.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:29 smithi189 ceph-mon[36655]: pgmap v431: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 126 MiB/s wr, 14.52k op/s 2024-03-16T20:38:29.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:29 smithi023 ceph-mon[31879]: pgmap v431: 129 pgs: 129 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 126 MiB/s wr, 14.52k op/s 2024-03-16T20:38:31.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:31 smithi031 ceph-mon[36281]: pgmap v432: 129 pgs: 129 active+clean; 13 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 116 MiB/s wr, 13.39k op/s 2024-03-16T20:38:31.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:31 smithi189 ceph-mon[36655]: pgmap v432: 129 pgs: 129 active+clean; 13 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 116 MiB/s wr, 13.39k op/s 2024-03-16T20:38:31.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:31 smithi023 ceph-mon[31879]: pgmap v432: 129 pgs: 129 active+clean; 13 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 116 MiB/s wr, 13.39k op/s 2024-03-16T20:38:33.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:33 smithi031 ceph-mon[36281]: pgmap v433: 129 pgs: 129 active+clean; 13 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 148 MiB/s wr, 16.98k op/s 2024-03-16T20:38:33.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:33 smithi189 ceph-mon[36655]: pgmap v433: 129 pgs: 129 active+clean; 13 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 148 MiB/s wr, 16.98k op/s 2024-03-16T20:38:33.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:33 smithi023 ceph-mon[31879]: pgmap v433: 129 pgs: 129 active+clean; 13 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 148 MiB/s wr, 16.98k op/s 2024-03-16T20:38:35.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:35 smithi031 ceph-mon[36281]: pgmap v434: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 121 MiB/s wr, 13.96k op/s 2024-03-16T20:38:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:35 smithi189 ceph-mon[36655]: pgmap v434: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 121 MiB/s wr, 13.96k op/s 2024-03-16T20:38:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:35 smithi023 ceph-mon[31879]: pgmap v434: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 121 MiB/s wr, 13.96k op/s 2024-03-16T20:38:37.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:37 smithi031 ceph-mon[36281]: pgmap v435: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 130 MiB/s wr, 14.93k op/s 2024-03-16T20:38:37.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:37 smithi189 ceph-mon[36655]: pgmap v435: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 130 MiB/s wr, 14.93k op/s 2024-03-16T20:38:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:37 smithi023 ceph-mon[31879]: pgmap v435: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 130 MiB/s wr, 14.93k op/s 2024-03-16T20:38:39.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:39 smithi031 ceph-mon[36281]: pgmap v436: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 138 MiB/s wr, 16.01k op/s 2024-03-16T20:38:39.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:39 smithi189 ceph-mon[36655]: pgmap v436: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 138 MiB/s wr, 16.01k op/s 2024-03-16T20:38:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:39 smithi023 ceph-mon[31879]: pgmap v436: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 138 MiB/s wr, 16.01k op/s 2024-03-16T20:38:41.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:41 smithi031 ceph-mon[36281]: pgmap v437: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 111 MiB/s wr, 12.85k op/s 2024-03-16T20:38:41.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:41 smithi189 ceph-mon[36655]: pgmap v437: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 111 MiB/s wr, 12.85k op/s 2024-03-16T20:38:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:41 smithi023 ceph-mon[31879]: pgmap v437: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 111 MiB/s wr, 12.85k op/s 2024-03-16T20:38:43.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:43 smithi031 ceph-mon[36281]: pgmap v438: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 131 MiB/s wr, 15.43k op/s 2024-03-16T20:38:43.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:43 smithi189 ceph-mon[36655]: pgmap v438: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 131 MiB/s wr, 15.43k op/s 2024-03-16T20:38:43.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:43 smithi023 ceph-mon[31879]: pgmap v438: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 131 MiB/s wr, 15.43k op/s 2024-03-16T20:38:45.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:45 smithi031 ceph-mon[36281]: pgmap v439: 129 pgs: 129 active+clean; 13 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 103 MiB/s wr, 12.48k op/s 2024-03-16T20:38:45.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:45 smithi189 ceph-mon[36655]: pgmap v439: 129 pgs: 129 active+clean; 13 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 103 MiB/s wr, 12.48k op/s 2024-03-16T20:38:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:45 smithi023 ceph-mon[31879]: pgmap v439: 129 pgs: 129 active+clean; 13 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 103 MiB/s wr, 12.48k op/s 2024-03-16T20:38:47.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:47 smithi031 ceph-mon[36281]: pgmap v440: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 101 MiB/s wr, 13.03k op/s 2024-03-16T20:38:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:47 smithi189 ceph-mon[36655]: pgmap v440: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 101 MiB/s wr, 13.03k op/s 2024-03-16T20:38:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:47 smithi023 ceph-mon[31879]: pgmap v440: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 101 MiB/s wr, 13.03k op/s 2024-03-16T20:38:49.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:49 smithi031 ceph-mon[36281]: pgmap v441: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 92 MiB/s wr, 12.55k op/s 2024-03-16T20:38:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:49 smithi189 ceph-mon[36655]: pgmap v441: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 92 MiB/s wr, 12.55k op/s 2024-03-16T20:38:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:49 smithi023 ceph-mon[31879]: pgmap v441: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 92 MiB/s wr, 12.55k op/s 2024-03-16T20:38:51.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:51 smithi031 ceph-mon[36281]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 67 MiB/s wr, 9.79k op/s 2024-03-16T20:38:51.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:51 smithi189 ceph-mon[36655]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 67 MiB/s wr, 9.79k op/s 2024-03-16T20:38:51.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:51 smithi023 ceph-mon[31879]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 67 MiB/s wr, 9.79k op/s 2024-03-16T20:38:53.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:53 smithi189 ceph-mon[36655]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 74 MiB/s wr, 11.48k op/s 2024-03-16T20:38:53.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:53 smithi023 ceph-mon[31879]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 74 MiB/s wr, 11.48k op/s 2024-03-16T20:38:53.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:53 smithi031 ceph-mon[36281]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 74 MiB/s wr, 11.48k op/s 2024-03-16T20:38:55.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:55 smithi189 ceph-mon[36655]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 51 MiB/s wr, 8.72k op/s 2024-03-16T20:38:55.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:55 smithi023 ceph-mon[31879]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 51 MiB/s wr, 8.72k op/s 2024-03-16T20:38:55.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:55 smithi031 ceph-mon[36281]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 51 MiB/s wr, 8.72k op/s 2024-03-16T20:38:57.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:57 smithi189 ceph-mon[36655]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 51 MiB/s wr, 9.19k op/s 2024-03-16T20:38:57.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:57 smithi023 ceph-mon[31879]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 51 MiB/s wr, 9.19k op/s 2024-03-16T20:38:57.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:57 smithi031 ceph-mon[36281]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 51 MiB/s wr, 9.19k op/s 2024-03-16T20:38:59.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:38:59 smithi189 ceph-mon[36655]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 50 MiB/s wr, 9.04k op/s 2024-03-16T20:38:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:38:59 smithi023 ceph-mon[31879]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 50 MiB/s wr, 9.04k op/s 2024-03-16T20:38:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:38:59 smithi031 ceph-mon[36281]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 50 MiB/s wr, 9.04k op/s 2024-03-16T20:39:01.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:01 smithi189 ceph-mon[36655]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 44 MiB/s wr, 8.21k op/s 2024-03-16T20:39:01.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:01 smithi023 ceph-mon[31879]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 44 MiB/s wr, 8.21k op/s 2024-03-16T20:39:01.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:01 smithi031 ceph-mon[36281]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 44 MiB/s wr, 8.21k op/s 2024-03-16T20:39:03.471 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:03 smithi189 ceph-mon[36655]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 52 MiB/s wr, 9.74k op/s 2024-03-16T20:39:03.489 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:03 smithi031 ceph-mon[36281]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 52 MiB/s wr, 9.74k op/s 2024-03-16T20:39:03.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:03 smithi023 ceph-mon[31879]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 52 MiB/s wr, 9.74k op/s 2024-03-16T20:39:05.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:05 smithi189 ceph-mon[36655]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 43 MiB/s wr, 8.00k op/s 2024-03-16T20:39:05.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:05 smithi023 ceph-mon[31879]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 43 MiB/s wr, 8.00k op/s 2024-03-16T20:39:05.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:05 smithi031 ceph-mon[36281]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 43 MiB/s wr, 8.00k op/s 2024-03-16T20:39:07.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:07 smithi189 ceph-mon[36655]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 48 MiB/s wr, 8.91k op/s 2024-03-16T20:39:07.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:07 smithi023 ceph-mon[31879]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 48 MiB/s wr, 8.91k op/s 2024-03-16T20:39:07.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:07 smithi031 ceph-mon[36281]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 48 MiB/s wr, 8.91k op/s 2024-03-16T20:39:09.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:09 smithi023 ceph-mon[31879]: pgmap v451: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 46 MiB/s wr, 8.61k op/s 2024-03-16T20:39:09.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:09 smithi031 ceph-mon[36281]: pgmap v451: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 46 MiB/s wr, 8.61k op/s 2024-03-16T20:39:09.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:09 smithi189 ceph-mon[36655]: pgmap v451: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 46 MiB/s wr, 8.61k op/s 2024-03-16T20:39:11.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:11 smithi023 ceph-mon[31879]: pgmap v452: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.27k op/s 2024-03-16T20:39:11.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:11 smithi031 ceph-mon[36281]: pgmap v452: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.27k op/s 2024-03-16T20:39:11.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:11 smithi189 ceph-mon[36655]: pgmap v452: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.27k op/s 2024-03-16T20:39:13.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:13 smithi023 ceph-mon[31879]: pgmap v453: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 7.99k op/s 2024-03-16T20:39:13.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:13 smithi031 ceph-mon[36281]: pgmap v453: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 7.99k op/s 2024-03-16T20:39:13.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:13 smithi189 ceph-mon[36655]: pgmap v453: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 42 MiB/s wr, 7.99k op/s 2024-03-16T20:39:15.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:15 smithi023 ceph-mon[31879]: pgmap v454: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 36 MiB/s wr, 7.08k op/s 2024-03-16T20:39:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:15 smithi031 ceph-mon[36281]: pgmap v454: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 36 MiB/s wr, 7.08k op/s 2024-03-16T20:39:15.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:15 smithi189 ceph-mon[36655]: pgmap v454: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 36 MiB/s wr, 7.08k op/s 2024-03-16T20:39:17.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:17 smithi023 ceph-mon[31879]: pgmap v455: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 44 MiB/s wr, 8.53k op/s 2024-03-16T20:39:17.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:17 smithi031 ceph-mon[36281]: pgmap v455: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 44 MiB/s wr, 8.53k op/s 2024-03-16T20:39:17.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:17 smithi189 ceph-mon[36655]: pgmap v455: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 44 MiB/s wr, 8.53k op/s 2024-03-16T20:39:19.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:19 smithi023 ceph-mon[31879]: pgmap v456: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 43 MiB/s wr, 8.63k op/s 2024-03-16T20:39:19.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:19 smithi031 ceph-mon[36281]: pgmap v456: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 43 MiB/s wr, 8.63k op/s 2024-03-16T20:39:19.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:19 smithi189 ceph-mon[36655]: pgmap v456: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 43 MiB/s wr, 8.63k op/s 2024-03-16T20:39:21.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:21 smithi023 ceph-mon[31879]: pgmap v457: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 38 MiB/s wr, 7.57k op/s 2024-03-16T20:39:21.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:21 smithi031 ceph-mon[36281]: pgmap v457: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 38 MiB/s wr, 7.57k op/s 2024-03-16T20:39:21.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:21 smithi189 ceph-mon[36655]: pgmap v457: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 38 MiB/s wr, 7.57k op/s 2024-03-16T20:39:23.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:23 smithi023 ceph-mon[31879]: pgmap v458: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 44 MiB/s wr, 8.94k op/s 2024-03-16T20:39:23.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:23 smithi031 ceph-mon[36281]: pgmap v458: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 44 MiB/s wr, 8.94k op/s 2024-03-16T20:39:23.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:23 smithi189 ceph-mon[36655]: pgmap v458: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 44 MiB/s wr, 8.94k op/s 2024-03-16T20:39:25.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:25 smithi023 ceph-mon[31879]: pgmap v459: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 40 MiB/s wr, 8.43k op/s 2024-03-16T20:39:25.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:25 smithi031 ceph-mon[36281]: pgmap v459: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 40 MiB/s wr, 8.43k op/s 2024-03-16T20:39:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:25 smithi189 ceph-mon[36655]: pgmap v459: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 40 MiB/s wr, 8.43k op/s 2024-03-16T20:39:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:27 smithi023 ceph-mon[31879]: pgmap v460: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 37 MiB/s wr, 7.84k op/s 2024-03-16T20:39:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:27 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:39:27.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:27 smithi031 ceph-mon[36281]: pgmap v460: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 37 MiB/s wr, 7.84k op/s 2024-03-16T20:39:27.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:27 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:39:27.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:27 smithi189 ceph-mon[36655]: pgmap v460: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 37 MiB/s wr, 7.84k op/s 2024-03-16T20:39:27.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:27 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:39:29.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:29 smithi023 ceph-mon[31879]: pgmap v461: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-03-16T20:39:29.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:29 smithi031 ceph-mon[36281]: pgmap v461: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-03-16T20:39:29.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:29 smithi189 ceph-mon[36655]: pgmap v461: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 33 MiB/s wr, 7.08k op/s 2024-03-16T20:39:31.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:31 smithi023 ceph-mon[31879]: pgmap v462: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.16k op/s 2024-03-16T20:39:31.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:31 smithi031 ceph-mon[36281]: pgmap v462: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.16k op/s 2024-03-16T20:39:31.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:31 smithi189 ceph-mon[36655]: pgmap v462: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.16k op/s 2024-03-16T20:39:33.521 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:33 smithi189 ceph-mon[36655]: pgmap v463: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 6.08k op/s 2024-03-16T20:39:33.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:33 smithi023 ceph-mon[31879]: pgmap v463: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 6.08k op/s 2024-03-16T20:39:33.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:33 smithi031 ceph-mon[36281]: pgmap v463: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 6.08k op/s 2024-03-16T20:39:35.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:35 smithi023 ceph-mon[31879]: pgmap v464: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 32 MiB/s wr, 5.52k op/s 2024-03-16T20:39:35.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:35 smithi031 ceph-mon[36281]: pgmap v464: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 32 MiB/s wr, 5.52k op/s 2024-03-16T20:39:35.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:35 smithi189 ceph-mon[36655]: pgmap v464: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 32 MiB/s wr, 5.52k op/s 2024-03-16T20:39:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:37 smithi023 ceph-mon[31879]: pgmap v465: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 5.02k op/s 2024-03-16T20:39:37.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:37 smithi031 ceph-mon[36281]: pgmap v465: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 5.02k op/s 2024-03-16T20:39:37.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:37 smithi189 ceph-mon[36655]: pgmap v465: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 5.02k op/s 2024-03-16T20:39:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:39 smithi023 ceph-mon[31879]: pgmap v466: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 38 MiB/s wr, 5.37k op/s 2024-03-16T20:39:39.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:39 smithi031 ceph-mon[36281]: pgmap v466: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 38 MiB/s wr, 5.37k op/s 2024-03-16T20:39:39.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:39 smithi189 ceph-mon[36655]: pgmap v466: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 38 MiB/s wr, 5.37k op/s 2024-03-16T20:39:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:41 smithi023 ceph-mon[31879]: pgmap v467: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 42 MiB/s wr, 4.99k op/s 2024-03-16T20:39:41.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:41 smithi031 ceph-mon[36281]: pgmap v467: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 42 MiB/s wr, 4.99k op/s 2024-03-16T20:39:41.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:41 smithi189 ceph-mon[36655]: pgmap v467: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 42 MiB/s wr, 4.99k op/s 2024-03-16T20:39:43.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:43 smithi023 ceph-mon[31879]: pgmap v468: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 46 MiB/s wr, 5.16k op/s 2024-03-16T20:39:43.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:43 smithi031 ceph-mon[36281]: pgmap v468: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 46 MiB/s wr, 5.16k op/s 2024-03-16T20:39:43.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:43 smithi189 ceph-mon[36655]: pgmap v468: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 46 MiB/s wr, 5.16k op/s 2024-03-16T20:39:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:45 smithi023 ceph-mon[31879]: pgmap v469: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 59 MiB/s wr, 6.54k op/s 2024-03-16T20:39:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:39:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:39:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:39:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:45 smithi031 ceph-mon[36281]: pgmap v469: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 59 MiB/s wr, 6.54k op/s 2024-03-16T20:39:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:39:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:39:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:39:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:45 smithi189 ceph-mon[36655]: pgmap v469: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 59 MiB/s wr, 6.54k op/s 2024-03-16T20:39:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:39:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:39:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:39:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:47 smithi023 ceph-mon[31879]: pgmap v470: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 61 MiB/s wr, 6.60k op/s 2024-03-16T20:39:47.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:47 smithi031 ceph-mon[36281]: pgmap v470: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 61 MiB/s wr, 6.60k op/s 2024-03-16T20:39:47.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:47 smithi189 ceph-mon[36655]: pgmap v470: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 61 MiB/s wr, 6.60k op/s 2024-03-16T20:39:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:49 smithi023 ceph-mon[31879]: pgmap v471: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 62 MiB/s wr, 6.86k op/s 2024-03-16T20:39:49.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:49 smithi031 ceph-mon[36281]: pgmap v471: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 62 MiB/s wr, 6.86k op/s 2024-03-16T20:39:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:49 smithi189 ceph-mon[36655]: pgmap v471: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 62 MiB/s wr, 6.86k op/s 2024-03-16T20:39:51.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:51 smithi023 ceph-mon[31879]: pgmap v472: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 63 MiB/s wr, 7.32k op/s 2024-03-16T20:39:51.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:51 smithi031 ceph-mon[36281]: pgmap v472: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 63 MiB/s wr, 7.32k op/s 2024-03-16T20:39:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:51 smithi189 ceph-mon[36655]: pgmap v472: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 63 MiB/s wr, 7.32k op/s 2024-03-16T20:39:52.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:52 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:52 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:52 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:53.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:53 smithi023 ceph-mon[31879]: pgmap v473: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 58 MiB/s wr, 7.47k op/s 2024-03-16T20:39:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:53 smithi031 ceph-mon[36281]: pgmap v473: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 58 MiB/s wr, 7.47k op/s 2024-03-16T20:39:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:53 smithi189 ceph-mon[36655]: pgmap v473: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 58 MiB/s wr, 7.47k op/s 2024-03-16T20:39:55.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:55 smithi023 ceph-mon[31879]: pgmap v474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 7.71k op/s 2024-03-16T20:39:55.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:55 smithi031 ceph-mon[36281]: pgmap v474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 7.71k op/s 2024-03-16T20:39:55.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:55 smithi189 ceph-mon[36655]: pgmap v474: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 7.71k op/s 2024-03-16T20:39:57.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:57 smithi023 ceph-mon[31879]: pgmap v475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 43 MiB/s wr, 6.71k op/s 2024-03-16T20:39:57.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:57 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:57.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:57 smithi031 ceph-mon[36281]: pgmap v475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 43 MiB/s wr, 6.71k op/s 2024-03-16T20:39:57.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:57 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:57 smithi189 ceph-mon[36655]: pgmap v475: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 43 MiB/s wr, 6.71k op/s 2024-03-16T20:39:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:57 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:39:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:39:59 smithi023 ceph-mon[31879]: pgmap v476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 MiB/s wr, 5.82k op/s 2024-03-16T20:39:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:39:59 smithi031 ceph-mon[36281]: pgmap v476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 MiB/s wr, 5.82k op/s 2024-03-16T20:39:59.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:39:59 smithi189 ceph-mon[36655]: pgmap v476: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 33 MiB/s wr, 5.82k op/s 2024-03-16T20:40:00.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:00 smithi031 ceph-mon[36281]: overall HEALTH_OK 2024-03-16T20:40:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:00 smithi189 ceph-mon[36655]: overall HEALTH_OK 2024-03-16T20:40:00.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:00 smithi023 ceph-mon[31879]: overall HEALTH_OK 2024-03-16T20:40:01.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:01 smithi031 ceph-mon[36281]: pgmap v477: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 6.83k op/s 2024-03-16T20:40:01.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:01 smithi189 ceph-mon[36655]: pgmap v477: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 6.83k op/s 2024-03-16T20:40:01.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:01 smithi023 ceph-mon[31879]: pgmap v477: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 6.83k op/s 2024-03-16T20:40:03.640 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:03 smithi023 ceph-mon[31879]: pgmap v478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 31 MiB/s wr, 6.28k op/s 2024-03-16T20:40:03.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:03 smithi031 ceph-mon[36281]: pgmap v478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 31 MiB/s wr, 6.28k op/s 2024-03-16T20:40:03.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:03 smithi189 ceph-mon[36655]: pgmap v478: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 31 MiB/s wr, 6.28k op/s 2024-03-16T20:40:05.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:05 smithi031 ceph-mon[36281]: pgmap v479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 33 MiB/s wr, 6.27k op/s 2024-03-16T20:40:05.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:05 smithi189 ceph-mon[36655]: pgmap v479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 33 MiB/s wr, 6.27k op/s 2024-03-16T20:40:05.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:05 smithi023 ceph-mon[31879]: pgmap v479: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 33 MiB/s wr, 6.27k op/s 2024-03-16T20:40:07.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:07 smithi031 ceph-mon[36281]: pgmap v480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 MiB/s wr, 6.85k op/s 2024-03-16T20:40:07.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:07 smithi189 ceph-mon[36655]: pgmap v480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 MiB/s wr, 6.85k op/s 2024-03-16T20:40:07.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:07 smithi023 ceph-mon[31879]: pgmap v480: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 MiB/s wr, 6.85k op/s 2024-03-16T20:40:09.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:09 smithi031 ceph-mon[36281]: pgmap v481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 34 MiB/s wr, 6.43k op/s 2024-03-16T20:40:09.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:09 smithi189 ceph-mon[36655]: pgmap v481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 34 MiB/s wr, 6.43k op/s 2024-03-16T20:40:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:09 smithi023 ceph-mon[31879]: pgmap v481: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 34 MiB/s wr, 6.43k op/s 2024-03-16T20:40:11.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:11 smithi031 ceph-mon[36281]: pgmap v482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 47 MiB/s wr, 8.15k op/s 2024-03-16T20:40:11.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:11 smithi189 ceph-mon[36655]: pgmap v482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 47 MiB/s wr, 8.15k op/s 2024-03-16T20:40:11.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:11 smithi023 ceph-mon[31879]: pgmap v482: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 47 MiB/s wr, 8.15k op/s 2024-03-16T20:40:13.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:13 smithi031 ceph-mon[36281]: pgmap v483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-03-16T20:40:13.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:13 smithi189 ceph-mon[36655]: pgmap v483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-03-16T20:40:13.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:13 smithi023 ceph-mon[31879]: pgmap v483: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-03-16T20:40:15.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:15 smithi031 ceph-mon[36281]: pgmap v484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 48 MiB/s wr, 7.25k op/s 2024-03-16T20:40:15.856 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:15 smithi023 ceph-mon[31879]: pgmap v484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 48 MiB/s wr, 7.25k op/s 2024-03-16T20:40:15.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:15 smithi189 ceph-mon[36655]: pgmap v484: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 48 MiB/s wr, 7.25k op/s 2024-03-16T20:40:17.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:17 smithi031 ceph-mon[36281]: pgmap v485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 MiB/s wr, 8.45k op/s 2024-03-16T20:40:17.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:17 smithi189 ceph-mon[36655]: pgmap v485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 MiB/s wr, 8.45k op/s 2024-03-16T20:40:17.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:17 smithi023 ceph-mon[31879]: pgmap v485: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 59 MiB/s wr, 8.45k op/s 2024-03-16T20:40:19.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:19 smithi031 ceph-mon[36281]: pgmap v486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-16T20:40:19.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:19 smithi189 ceph-mon[36655]: pgmap v486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-16T20:40:19.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:19 smithi023 ceph-mon[31879]: pgmap v486: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-16T20:40:21.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:21 smithi031 ceph-mon[36281]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 MiB/s wr, 9.16k op/s 2024-03-16T20:40:21.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:21 smithi189 ceph-mon[36655]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 MiB/s wr, 9.16k op/s 2024-03-16T20:40:21.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:21 smithi023 ceph-mon[31879]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 MiB/s wr, 9.16k op/s 2024-03-16T20:40:23.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:23 smithi031 ceph-mon[36281]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 57 MiB/s wr, 7.93k op/s 2024-03-16T20:40:23.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:23 smithi189 ceph-mon[36655]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 57 MiB/s wr, 7.93k op/s 2024-03-16T20:40:23.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:23 smithi023 ceph-mon[31879]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 57 MiB/s wr, 7.93k op/s 2024-03-16T20:40:25.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:25 smithi031 ceph-mon[36281]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 54 MiB/s wr, 7.65k op/s 2024-03-16T20:40:25.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:25 smithi189 ceph-mon[36655]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 54 MiB/s wr, 7.65k op/s 2024-03-16T20:40:25.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:25 smithi023 ceph-mon[31879]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 54 MiB/s wr, 7.65k op/s 2024-03-16T20:40:27.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:27 smithi031 ceph-mon[36281]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 58 MiB/s wr, 8.25k op/s 2024-03-16T20:40:27.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:27 smithi189 ceph-mon[36655]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 58 MiB/s wr, 8.25k op/s 2024-03-16T20:40:27.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:27 smithi023 ceph-mon[31879]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 58 MiB/s wr, 8.25k op/s 2024-03-16T20:40:29.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:29 smithi189 ceph-mon[36655]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 42 MiB/s wr, 6.36k op/s 2024-03-16T20:40:29.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:29 smithi023 ceph-mon[31879]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 42 MiB/s wr, 6.36k op/s 2024-03-16T20:40:29.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:29 smithi031 ceph-mon[36281]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 42 MiB/s wr, 6.36k op/s 2024-03-16T20:40:31.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:31 smithi189 ceph-mon[36655]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 50 MiB/s wr, 7.41k op/s 2024-03-16T20:40:31.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:31 smithi023 ceph-mon[31879]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 50 MiB/s wr, 7.41k op/s 2024-03-16T20:40:31.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:31 smithi031 ceph-mon[36281]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 50 MiB/s wr, 7.41k op/s 2024-03-16T20:40:33.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:33 smithi189 ceph-mon[36655]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 45 MiB/s wr, 6.83k op/s 2024-03-16T20:40:33.738 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:33 smithi031 ceph-mon[36281]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 45 MiB/s wr, 6.83k op/s 2024-03-16T20:40:33.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:33 smithi023 ceph-mon[31879]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 45 MiB/s wr, 6.83k op/s 2024-03-16T20:40:35.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:35 smithi189 ceph-mon[36655]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 41 MiB/s wr, 6.28k op/s 2024-03-16T20:40:35.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:35 smithi023 ceph-mon[31879]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 41 MiB/s wr, 6.28k op/s 2024-03-16T20:40:35.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:35 smithi031 ceph-mon[36281]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 41 MiB/s wr, 6.28k op/s 2024-03-16T20:40:37.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:37 smithi189 ceph-mon[36655]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 45 MiB/s wr, 7.14k op/s 2024-03-16T20:40:37.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:37 smithi023 ceph-mon[31879]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 45 MiB/s wr, 7.14k op/s 2024-03-16T20:40:37.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:37 smithi031 ceph-mon[36281]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 45 MiB/s wr, 7.14k op/s 2024-03-16T20:40:39.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:39 smithi189 ceph-mon[36655]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 39 MiB/s wr, 6.57k op/s 2024-03-16T20:40:39.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:39 smithi023 ceph-mon[31879]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 39 MiB/s wr, 6.57k op/s 2024-03-16T20:40:39.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:39 smithi031 ceph-mon[36281]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 39 MiB/s wr, 6.57k op/s 2024-03-16T20:40:41.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:41 smithi189 ceph-mon[36655]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 41 MiB/s wr, 7.08k op/s 2024-03-16T20:40:41.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:41 smithi023 ceph-mon[31879]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 41 MiB/s wr, 7.08k op/s 2024-03-16T20:40:41.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:41 smithi031 ceph-mon[36281]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 41 MiB/s wr, 7.08k op/s 2024-03-16T20:40:43.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:43 smithi023 ceph-mon[31879]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 41 MiB/s wr, 7.43k op/s 2024-03-16T20:40:43.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:43 smithi031 ceph-mon[36281]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 41 MiB/s wr, 7.43k op/s 2024-03-16T20:40:43.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:43 smithi189 ceph-mon[36655]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 41 MiB/s wr, 7.43k op/s 2024-03-16T20:40:45.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:45 smithi023 ceph-mon[31879]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:40:45.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:45 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:40:45.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:45 smithi031 ceph-mon[36281]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:40:45.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:45 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:40:46.060 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:45 smithi189 ceph-mon[36655]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:40:46.061 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:45 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:40:47.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:47 smithi023 ceph-mon[31879]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 7.15k op/s 2024-03-16T20:40:47.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:47 smithi031 ceph-mon[36281]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 7.15k op/s 2024-03-16T20:40:48.089 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:47 smithi189 ceph-mon[36655]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 7.15k op/s 2024-03-16T20:40:48.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:40:48.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:40:48.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:40:48.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:48 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:40:48.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:48 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:40:48.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:48 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:40:48.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:40:48.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:40:48.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:40:49.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:49 smithi023 ceph-mon[31879]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 7.29k op/s 2024-03-16T20:40:49.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:49 smithi031 ceph-mon[36281]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 7.29k op/s 2024-03-16T20:40:49.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:49 smithi189 ceph-mon[36655]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 7.29k op/s 2024-03-16T20:40:51.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:51 smithi023 ceph-mon[31879]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 35 MiB/s wr, 6.74k op/s 2024-03-16T20:40:51.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:51 smithi031 ceph-mon[36281]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 35 MiB/s wr, 6.74k op/s 2024-03-16T20:40:51.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:51 smithi189 ceph-mon[36655]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 35 MiB/s wr, 6.74k op/s 2024-03-16T20:40:53.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:53 smithi023 ceph-mon[31879]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 37 MiB/s wr, 7.11k op/s 2024-03-16T20:40:53.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:53 smithi031 ceph-mon[36281]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 37 MiB/s wr, 7.11k op/s 2024-03-16T20:40:53.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:53 smithi189 ceph-mon[36655]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 37 MiB/s wr, 7.11k op/s 2024-03-16T20:40:55.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:55 smithi023 ceph-mon[31879]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 31 MiB/s wr, 6.03k op/s 2024-03-16T20:40:55.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:55 smithi031 ceph-mon[36281]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 31 MiB/s wr, 6.03k op/s 2024-03-16T20:40:55.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:55 smithi189 ceph-mon[36655]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 31 MiB/s wr, 6.03k op/s 2024-03-16T20:40:57.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:57 smithi023 ceph-mon[31879]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 34 MiB/s wr, 6.74k op/s 2024-03-16T20:40:57.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:57 smithi031 ceph-mon[36281]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 34 MiB/s wr, 6.74k op/s 2024-03-16T20:40:57.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:57 smithi189 ceph-mon[36655]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 34 MiB/s wr, 6.74k op/s 2024-03-16T20:40:59.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:40:59 smithi023 ceph-mon[31879]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 35 MiB/s wr, 6.86k op/s 2024-03-16T20:40:59.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:40:59 smithi031 ceph-mon[36281]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 35 MiB/s wr, 6.86k op/s 2024-03-16T20:40:59.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:40:59 smithi189 ceph-mon[36655]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 35 MiB/s wr, 6.86k op/s 2024-03-16T20:41:01.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:01 smithi023 ceph-mon[31879]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 28 MiB/s wr, 5.69k op/s 2024-03-16T20:41:01.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:01 smithi031 ceph-mon[36281]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 28 MiB/s wr, 5.69k op/s 2024-03-16T20:41:01.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:01 smithi189 ceph-mon[36655]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 28 MiB/s wr, 5.69k op/s 2024-03-16T20:41:03.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:03 smithi023 ceph-mon[31879]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 33 MiB/s wr, 6.67k op/s 2024-03-16T20:41:03.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:03 smithi031 ceph-mon[36281]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 33 MiB/s wr, 6.67k op/s 2024-03-16T20:41:03.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:03 smithi189 ceph-mon[36655]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 33 MiB/s wr, 6.67k op/s 2024-03-16T20:41:05.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:05 smithi023 ceph-mon[31879]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 5.87k op/s 2024-03-16T20:41:05.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:05 smithi031 ceph-mon[36281]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 5.87k op/s 2024-03-16T20:41:05.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:05 smithi189 ceph-mon[36655]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 29 MiB/s wr, 5.87k op/s 2024-03-16T20:41:07.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:07 smithi023 ceph-mon[31879]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 29 MiB/s wr, 5.84k op/s 2024-03-16T20:41:07.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:07 smithi031 ceph-mon[36281]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 29 MiB/s wr, 5.84k op/s 2024-03-16T20:41:07.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:07 smithi189 ceph-mon[36655]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 29 MiB/s wr, 5.84k op/s 2024-03-16T20:41:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:09 smithi023 ceph-mon[31879]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 32 MiB/s wr, 6.38k op/s 2024-03-16T20:41:09.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:09 smithi031 ceph-mon[36281]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 32 MiB/s wr, 6.38k op/s 2024-03-16T20:41:09.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:09 smithi189 ceph-mon[36655]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 32 MiB/s wr, 6.38k op/s 2024-03-16T20:41:11.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:11 smithi023 ceph-mon[31879]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 29 MiB/s wr, 5.72k op/s 2024-03-16T20:41:11.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:11 smithi031 ceph-mon[36281]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 29 MiB/s wr, 5.72k op/s 2024-03-16T20:41:11.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:11 smithi189 ceph-mon[36655]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 29 MiB/s wr, 5.72k op/s 2024-03-16T20:41:13.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:13 smithi031 ceph-mon[36281]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 34 MiB/s wr, 6.56k op/s 2024-03-16T20:41:13.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:13 smithi189 ceph-mon[36655]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 34 MiB/s wr, 6.56k op/s 2024-03-16T20:41:14.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:13 smithi023 ceph-mon[31879]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 34 MiB/s wr, 6.56k op/s 2024-03-16T20:41:15.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:15 smithi031 ceph-mon[36281]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 29 MiB/s wr, 5.71k op/s 2024-03-16T20:41:15.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:15 smithi189 ceph-mon[36655]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 29 MiB/s wr, 5.71k op/s 2024-03-16T20:41:16.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:15 smithi023 ceph-mon[31879]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 29 MiB/s wr, 5.71k op/s 2024-03-16T20:41:17.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:17 smithi031 ceph-mon[36281]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 34 MiB/s wr, 6.62k op/s 2024-03-16T20:41:17.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:17 smithi189 ceph-mon[36655]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 34 MiB/s wr, 6.62k op/s 2024-03-16T20:41:18.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:17 smithi023 ceph-mon[31879]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 34 MiB/s wr, 6.62k op/s 2024-03-16T20:41:19.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:19 smithi031 ceph-mon[36281]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-16T20:41:19.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:19 smithi189 ceph-mon[36655]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-16T20:41:20.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:19 smithi023 ceph-mon[31879]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-16T20:41:21.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:21 smithi031 ceph-mon[36281]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 31 MiB/s wr, 6.01k op/s 2024-03-16T20:41:21.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:21 smithi189 ceph-mon[36655]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 31 MiB/s wr, 6.01k op/s 2024-03-16T20:41:22.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:21 smithi023 ceph-mon[31879]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 31 MiB/s wr, 6.01k op/s 2024-03-16T20:41:23.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:23 smithi031 ceph-mon[36281]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 35 MiB/s wr, 6.81k op/s 2024-03-16T20:41:23.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:23 smithi189 ceph-mon[36655]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 35 MiB/s wr, 6.81k op/s 2024-03-16T20:41:24.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:23 smithi023 ceph-mon[31879]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 35 MiB/s wr, 6.81k op/s 2024-03-16T20:41:25.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:25 smithi031 ceph-mon[36281]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-03-16T20:41:25.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:25 smithi189 ceph-mon[36655]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-03-16T20:41:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:25 smithi023 ceph-mon[31879]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 33 MiB/s wr, 6.31k op/s 2024-03-16T20:41:27.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:27 smithi031 ceph-mon[36281]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 34 MiB/s wr, 6.46k op/s 2024-03-16T20:41:27.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:27 smithi189 ceph-mon[36655]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 34 MiB/s wr, 6.46k op/s 2024-03-16T20:41:28.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:27 smithi023 ceph-mon[31879]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 34 MiB/s wr, 6.46k op/s 2024-03-16T20:41:29.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:29 smithi031 ceph-mon[36281]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 5.73k op/s 2024-03-16T20:41:29.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:29 smithi189 ceph-mon[36655]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 5.73k op/s 2024-03-16T20:41:30.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:29 smithi023 ceph-mon[31879]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 30 MiB/s wr, 5.73k op/s 2024-03-16T20:41:31.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:31 smithi031 ceph-mon[36281]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 29 MiB/s wr, 5.53k op/s 2024-03-16T20:41:31.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:31 smithi189 ceph-mon[36655]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 29 MiB/s wr, 5.53k op/s 2024-03-16T20:41:32.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:31 smithi023 ceph-mon[31879]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 29 MiB/s wr, 5.53k op/s 2024-03-16T20:41:33.891 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:33 smithi023 ceph-mon[31879]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 34 MiB/s wr, 6.51k op/s 2024-03-16T20:41:33.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:33 smithi031 ceph-mon[36281]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 34 MiB/s wr, 6.51k op/s 2024-03-16T20:41:33.971 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:33 smithi189 ceph-mon[36655]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 34 MiB/s wr, 6.51k op/s 2024-03-16T20:41:35.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:35 smithi031 ceph-mon[36281]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 26 MiB/s wr, 5.00k op/s 2024-03-16T20:41:35.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:35 smithi189 ceph-mon[36655]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 26 MiB/s wr, 5.00k op/s 2024-03-16T20:41:36.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:35 smithi023 ceph-mon[31879]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 26 MiB/s wr, 5.00k op/s 2024-03-16T20:41:37.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:37 smithi031 ceph-mon[36281]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 32 MiB/s wr, 6.11k op/s 2024-03-16T20:41:37.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:37 smithi189 ceph-mon[36655]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 32 MiB/s wr, 6.11k op/s 2024-03-16T20:41:38.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:37 smithi023 ceph-mon[31879]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 32 MiB/s wr, 6.11k op/s 2024-03-16T20:41:39.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:39 smithi031 ceph-mon[36281]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 31 MiB/s wr, 5.94k op/s 2024-03-16T20:41:39.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:39 smithi189 ceph-mon[36655]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 31 MiB/s wr, 5.94k op/s 2024-03-16T20:41:40.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:39 smithi023 ceph-mon[31879]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 31 MiB/s wr, 5.94k op/s 2024-03-16T20:41:40.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:40 smithi031 ceph-mon[36281]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-16T20:41:40.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:40 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:40.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:40 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:40.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:40 smithi189 ceph-mon[36655]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-16T20:41:40.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:40 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:40.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:40 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:41.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:40 smithi023 ceph-mon[31879]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-16T20:41:41.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:40 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:41.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:40 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:41.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:41 smithi031 ceph-mon[36281]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:41:41.970 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:41 smithi189 ceph-mon[36655]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:41:42.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:41 smithi023 ceph-mon[31879]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 32 MiB/s wr, 6.09k op/s 2024-03-16T20:41:43.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:42 smithi023 ceph-mon[31879]: Health check failed: 0 slow ops, oldest one blocked for 183 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:43.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:42 smithi031 ceph-mon[36281]: Health check failed: 0 slow ops, oldest one blocked for 183 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:43.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:42 smithi189 ceph-mon[36655]: Health check failed: 0 slow ops, oldest one blocked for 183 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:44.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:43 smithi023 ceph-mon[31879]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 38 MiB/s wr, 7.36k op/s 2024-03-16T20:41:44.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:43 smithi031 ceph-mon[36281]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 38 MiB/s wr, 7.36k op/s 2024-03-16T20:41:44.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:43 smithi189 ceph-mon[36655]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 38 MiB/s wr, 7.36k op/s 2024-03-16T20:41:46.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:45 smithi023 ceph-mon[31879]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-16T20:41:46.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:45 smithi031 ceph-mon[36281]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-16T20:41:46.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:45 smithi189 ceph-mon[36655]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-16T20:41:48.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:47 smithi023 ceph-mon[31879]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 7.13k op/s 2024-03-16T20:41:48.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:47 smithi031 ceph-mon[36281]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 7.13k op/s 2024-03-16T20:41:48.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:47 smithi189 ceph-mon[36655]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 7.13k op/s 2024-03-16T20:41:49.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:48 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:49.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:48 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:49.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:48 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:41:49.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:48 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:49.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:48 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:49.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:48 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:41:49.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:48 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:49.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:48 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:49.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:48 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:41:50.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:49 smithi023 ceph-mon[31879]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 32 MiB/s wr, 6.37k op/s 2024-03-16T20:41:50.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:49 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 188 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:49 smithi031 ceph-mon[36281]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 32 MiB/s wr, 6.37k op/s 2024-03-16T20:41:50.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:49 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 188 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:50.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:49 smithi189 ceph-mon[36655]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 32 MiB/s wr, 6.37k op/s 2024-03-16T20:41:50.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:49 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 188 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:51.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:41:51.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:41:51.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:50 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:41:51.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:41:51.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:41:51.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:50 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:41:51.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:41:51.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:41:51.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:50 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:41:52.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:51 smithi023 ceph-mon[31879]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 27 MiB/s wr, 5.45k op/s 2024-03-16T20:41:52.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:51 smithi031 ceph-mon[36281]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 27 MiB/s wr, 5.45k op/s 2024-03-16T20:41:52.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:51 smithi189 ceph-mon[36655]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 27 MiB/s wr, 5.45k op/s 2024-03-16T20:41:53.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:52 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:53.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:52 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:53.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:52 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:53.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:52 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:53.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:52 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:53.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:52 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:54.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:53 smithi023 ceph-mon[31879]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 7.27k op/s 2024-03-16T20:41:54.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:53 smithi031 ceph-mon[36281]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 7.27k op/s 2024-03-16T20:41:54.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:53 smithi189 ceph-mon[36655]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 37 MiB/s wr, 7.27k op/s 2024-03-16T20:41:55.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:54 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 193 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:55.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:54 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 193 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:55.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:54 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 193 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:41:56.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:55 smithi023 ceph-mon[31879]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 26 MiB/s wr, 5.16k op/s 2024-03-16T20:41:56.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:55 smithi031 ceph-mon[36281]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 26 MiB/s wr, 5.16k op/s 2024-03-16T20:41:56.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:55 smithi189 ceph-mon[36655]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 26 MiB/s wr, 5.16k op/s 2024-03-16T20:41:57.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:56 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:57.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:56 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:57.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:56 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:57.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:56 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:56 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:41:57.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:56 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:41:58.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:57 smithi031 ceph-mon[36281]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 31 MiB/s wr, 6.13k op/s 2024-03-16T20:41:58.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:57 smithi189 ceph-mon[36655]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 31 MiB/s wr, 6.13k op/s 2024-03-16T20:41:58.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:57 smithi023 ceph-mon[31879]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 31 MiB/s wr, 6.13k op/s 2024-03-16T20:42:00.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:59 smithi031 ceph-mon[36281]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 34 MiB/s wr, 6.61k op/s 2024-03-16T20:42:00.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:41:59 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:00.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:59 smithi189 ceph-mon[36655]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 34 MiB/s wr, 6.61k op/s 2024-03-16T20:42:00.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:41:59 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:59 smithi023 ceph-mon[31879]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 34 MiB/s wr, 6.61k op/s 2024-03-16T20:42:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:41:59 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 199 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:01.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:00 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:01.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:00 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:01.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:01.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:00 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:02.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:01 smithi031 ceph-mon[36281]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 29 MiB/s wr, 5.66k op/s 2024-03-16T20:42:02.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:01 smithi189 ceph-mon[36655]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 29 MiB/s wr, 5.66k op/s 2024-03-16T20:42:02.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:01 smithi023 ceph-mon[31879]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 29 MiB/s wr, 5.66k op/s 2024-03-16T20:42:04.140 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:03 smithi023 ceph-mon[31879]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 39 MiB/s wr, 7.74k op/s 2024-03-16T20:42:04.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:03 smithi031 ceph-mon[36281]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 39 MiB/s wr, 7.74k op/s 2024-03-16T20:42:04.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:03 smithi189 ceph-mon[36655]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 39 MiB/s wr, 7.74k op/s 2024-03-16T20:42:05.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:04 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 205 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:05.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:04 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 205 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:05.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:04 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 205 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:06.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:05 smithi031 ceph-mon[36281]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 28 MiB/s wr, 5.54k op/s 2024-03-16T20:42:06.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:05 smithi189 ceph-mon[36655]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 28 MiB/s wr, 5.54k op/s 2024-03-16T20:42:06.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:05 smithi023 ceph-mon[31879]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 28 MiB/s wr, 5.54k op/s 2024-03-16T20:42:08.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:07 smithi189 ceph-mon[36655]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 5.88k op/s 2024-03-16T20:42:08.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:07 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:08.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:07 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:08.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:07 smithi023 ceph-mon[31879]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 5.88k op/s 2024-03-16T20:42:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:07 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:07 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:08.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:07 smithi031 ceph-mon[36281]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 5.88k op/s 2024-03-16T20:42:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:07 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:08.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:07 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:10.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:09 smithi189 ceph-mon[36655]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 34 MiB/s wr, 6.81k op/s 2024-03-16T20:42:10.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:09 smithi023 ceph-mon[31879]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 34 MiB/s wr, 6.81k op/s 2024-03-16T20:42:10.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:09 smithi031 ceph-mon[36281]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 34 MiB/s wr, 6.81k op/s 2024-03-16T20:42:12.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:11 smithi189 ceph-mon[36655]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 25 MiB/s wr, 5.02k op/s 2024-03-16T20:42:12.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:11 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:12.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:11 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:12.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:11 smithi023 ceph-mon[31879]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 25 MiB/s wr, 5.02k op/s 2024-03-16T20:42:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:11 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:11 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:11 smithi031 ceph-mon[36281]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 25 MiB/s wr, 5.02k op/s 2024-03-16T20:42:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:11 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:12.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:11 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:14.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:13 smithi189 ceph-mon[36655]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 36 MiB/s wr, 7.30k op/s 2024-03-16T20:42:14.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:13 smithi023 ceph-mon[31879]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 36 MiB/s wr, 7.30k op/s 2024-03-16T20:42:14.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:13 smithi031 ceph-mon[36281]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 36 MiB/s wr, 7.30k op/s 2024-03-16T20:42:15.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:14 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 210 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:15.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:14 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 210 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:15.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:14 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 210 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:16.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:15 smithi023 ceph-mon[31879]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:15 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:15 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:15 smithi031 ceph-mon[36281]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:15 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:15 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:15 smithi189 ceph-mon[36655]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:15 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:15 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:18.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:17 smithi023 ceph-mon[31879]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 28 MiB/s wr, 5.73k op/s 2024-03-16T20:42:18.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:17 smithi031 ceph-mon[36281]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 28 MiB/s wr, 5.73k op/s 2024-03-16T20:42:18.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:17 smithi189 ceph-mon[36655]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 28 MiB/s wr, 5.73k op/s 2024-03-16T20:42:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:19 smithi023 ceph-mon[31879]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 33 MiB/s wr, 6.89k op/s 2024-03-16T20:42:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:19 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:19 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:19 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 215 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:19 smithi031 ceph-mon[36281]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 33 MiB/s wr, 6.89k op/s 2024-03-16T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:19 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:19 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:19 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 215 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:19 smithi189 ceph-mon[36655]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 33 MiB/s wr, 6.89k op/s 2024-03-16T20:42:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:19 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:19 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:19 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 215 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:22.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:21 smithi023 ceph-mon[31879]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 26 MiB/s wr, 5.39k op/s 2024-03-16T20:42:22.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:21 smithi031 ceph-mon[36281]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 26 MiB/s wr, 5.39k op/s 2024-03-16T20:42:22.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:21 smithi189 ceph-mon[36655]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 26 MiB/s wr, 5.39k op/s 2024-03-16T20:42:24.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:23 smithi023 ceph-mon[31879]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 35 MiB/s wr, 7.24k op/s 2024-03-16T20:42:24.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:23 smithi031 ceph-mon[36281]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 35 MiB/s wr, 7.24k op/s 2024-03-16T20:42:24.472 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:23 smithi189 ceph-mon[36655]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 35 MiB/s wr, 7.24k op/s 2024-03-16T20:42:25.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:25 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 221 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:25.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:25 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 221 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:25.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:25 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 221 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:26.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:26 smithi023 ceph-mon[31879]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 5.21k op/s 2024-03-16T20:42:26.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:26 smithi031 ceph-mon[36281]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 5.21k op/s 2024-03-16T20:42:26.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:26 smithi189 ceph-mon[36655]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 5.21k op/s 2024-03-16T20:42:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:28 smithi023 ceph-mon[31879]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-16T20:42:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:28 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:28.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:28 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:28.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:28 smithi031 ceph-mon[36281]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-16T20:42:28.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:28 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:28.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:28 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:28 smithi189 ceph-mon[36655]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-16T20:42:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:28 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:28 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:30 smithi023 ceph-mon[31879]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 34 MiB/s wr, 6.87k op/s 2024-03-16T20:42:30.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:30 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:30.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:30 smithi031 ceph-mon[36281]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 34 MiB/s wr, 6.87k op/s 2024-03-16T20:42:30.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:30 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:30.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:30 smithi189 ceph-mon[36655]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 34 MiB/s wr, 6.87k op/s 2024-03-16T20:42:30.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:30 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:32.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:32 smithi023 ceph-mon[31879]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 5.38k op/s 2024-03-16T20:42:32.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:32 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:32.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:32 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:32.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:32 smithi031 ceph-mon[36281]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 5.38k op/s 2024-03-16T20:42:32.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:32 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:32.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:32 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:32.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:32 smithi189 ceph-mon[36655]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 5.38k op/s 2024-03-16T20:42:32.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:32 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:32.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:32 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:34.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:34 smithi023 ceph-mon[31879]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 MiB/s wr, 6.33k op/s 2024-03-16T20:42:34.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:34 smithi031 ceph-mon[36281]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 MiB/s wr, 6.33k op/s 2024-03-16T20:42:34.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:34 smithi189 ceph-mon[36655]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 MiB/s wr, 6.33k op/s 2024-03-16T20:42:35.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:35 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 233 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:35.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:35 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 233 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:35.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:35 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 233 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:36.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:36 smithi023 ceph-mon[31879]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 27 MiB/s wr, 5.51k op/s 2024-03-16T20:42:36.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:36 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:36.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:36 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:36.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:36 smithi031 ceph-mon[36281]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 27 MiB/s wr, 5.51k op/s 2024-03-16T20:42:36.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:36 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:36.439 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:36 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:36.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:36 smithi189 ceph-mon[36655]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 27 MiB/s wr, 5.51k op/s 2024-03-16T20:42:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:36 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:36.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:36 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:38.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:38 smithi023 ceph-mon[31879]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 27 MiB/s wr, 5.49k op/s 2024-03-16T20:42:38.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:38 smithi031 ceph-mon[36281]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 27 MiB/s wr, 5.49k op/s 2024-03-16T20:42:38.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:38 smithi189 ceph-mon[36655]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 27 MiB/s wr, 5.49k op/s 2024-03-16T20:42:39.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:39 smithi031 ceph-mon[36281]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 33 MiB/s wr, 6.79k op/s 2024-03-16T20:42:39.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:39 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 238 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:39.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:39 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:39.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:39 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:39.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:39 smithi189 ceph-mon[36655]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 33 MiB/s wr, 6.79k op/s 2024-03-16T20:42:39.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:39 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 238 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:39.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:39 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:39.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:39 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:39.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:39 smithi023 ceph-mon[31879]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 33 MiB/s wr, 6.79k op/s 2024-03-16T20:42:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:39 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 238 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:39 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:39 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:41.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:41 smithi031 ceph-mon[36281]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:41.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:41 smithi189 ceph-mon[36655]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:41 smithi023 ceph-mon[31879]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:43.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:43 smithi031 ceph-mon[36281]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.00k op/s 2024-03-16T20:42:43.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:43 smithi189 ceph-mon[36655]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.00k op/s 2024-03-16T20:42:43.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:43 smithi023 ceph-mon[31879]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 29 MiB/s wr, 6.00k op/s 2024-03-16T20:42:44.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:44 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:44.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:44 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:44.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:44 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:45.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:45 smithi023 ceph-mon[31879]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:45 smithi031 ceph-mon[36281]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:45.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:45 smithi189 ceph-mon[36655]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 5.74k op/s 2024-03-16T20:42:47.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:47 smithi023 ceph-mon[31879]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2024-03-16T20:42:47.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:47 smithi031 ceph-mon[36281]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2024-03-16T20:42:47.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:47 smithi189 ceph-mon[36655]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 23 MiB/s wr, 4.74k op/s 2024-03-16T20:42:48.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:48 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:48.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:48 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:48.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:48 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:48.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:48 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:48 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:48.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:48 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:49 smithi023 ceph-mon[31879]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-03-16T20:42:49.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:49 smithi031 ceph-mon[36281]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-03-16T20:42:49.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:49 smithi189 ceph-mon[36655]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-03-16T20:42:50.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:50 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:50.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:50 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:50.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:50 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:42:51.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:51 smithi023 ceph-mon[31879]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:51.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:51 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:42:51.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:51 smithi031 ceph-mon[36281]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:51.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:51 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:42:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:51 smithi189 ceph-mon[36655]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-16T20:42:51.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:51 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: Unable to set osd_memory_target on smithi023 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-16T20:42:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi031 to 175.2M 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: Unable to set osd_memory_target on smithi031 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:42:52.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:42:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: Unable to set osd_memory_target on smithi023 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi031 to 175.2M 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: Unable to set osd_memory_target on smithi031 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:42:52.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: Unable to set osd_memory_target on smithi023 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi031 to 175.2M 2024-03-16T20:42:52.938 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: Unable to set osd_memory_target on smithi031 to 183744512: error parsing value: Value '183744512' is below minimum 939524096 2024-03-16T20:42:52.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:42:52.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:42:52.939 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:42:54.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:53 smithi189 ceph-mon[36655]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 29 MiB/s wr, 6.15k op/s 2024-03-16T20:42:54.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:53 smithi023 ceph-mon[31879]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 29 MiB/s wr, 6.15k op/s 2024-03-16T20:42:54.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:53 smithi031 ceph-mon[36281]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 29 MiB/s wr, 6.15k op/s 2024-03-16T20:42:56.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:55 smithi023 ceph-mon[31879]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 5.78k op/s 2024-03-16T20:42:56.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:55 smithi031 ceph-mon[36281]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 5.78k op/s 2024-03-16T20:42:56.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:55 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:56.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:55 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:56.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:55 smithi189 ceph-mon[36655]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 MiB/s wr, 5.78k op/s 2024-03-16T20:42:56.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:55 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:55 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:56.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:56 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:42:56.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:56 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:42:58.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:57 smithi023 ceph-mon[31879]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 5.29k op/s 2024-03-16T20:42:58.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:57 smithi031 ceph-mon[36281]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 5.29k op/s 2024-03-16T20:42:58.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:57 smithi189 ceph-mon[36655]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 25 MiB/s wr, 5.29k op/s 2024-03-16T20:43:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:59 smithi023 ceph-mon[31879]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 31 MiB/s wr, 6.57k op/s 2024-03-16T20:43:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:59 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:59 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:00.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:42:59 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:59 smithi031 ceph-mon[36281]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 31 MiB/s wr, 6.57k op/s 2024-03-16T20:43:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:59 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:59 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:00.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:42:59 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:00.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:59 smithi189 ceph-mon[36655]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 31 MiB/s wr, 6.57k op/s 2024-03-16T20:43:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:59 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:59 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:00.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:42:59 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:02.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:01 smithi023 ceph-mon[31879]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 28 MiB/s wr, 6.03k op/s 2024-03-16T20:43:02.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:01 smithi031 ceph-mon[36281]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 28 MiB/s wr, 6.03k op/s 2024-03-16T20:43:02.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:01 smithi189 ceph-mon[36655]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 28 MiB/s wr, 6.03k op/s 2024-03-16T20:43:04.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:04 smithi189 ceph-mon[36655]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 31 MiB/s wr, 6.49k op/s 2024-03-16T20:43:04.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:04 smithi023 ceph-mon[31879]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 31 MiB/s wr, 6.49k op/s 2024-03-16T20:43:04.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:04 smithi031 ceph-mon[36281]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 31 MiB/s wr, 6.49k op/s 2024-03-16T20:43:05.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:05 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:05 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:05.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:05 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:06.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:06 smithi023 ceph-mon[31879]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 5.50k op/s 2024-03-16T20:43:06.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:06 smithi031 ceph-mon[36281]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 5.50k op/s 2024-03-16T20:43:06.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:06 smithi189 ceph-mon[36655]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 5.50k op/s 2024-03-16T20:43:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:08 smithi023 ceph-mon[31879]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.04k op/s 2024-03-16T20:43:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:08 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:08.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:08 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:08.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:08 smithi031 ceph-mon[36281]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.04k op/s 2024-03-16T20:43:08.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:08 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:08.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:08 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:08.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:08 smithi189 ceph-mon[36655]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.04k op/s 2024-03-16T20:43:08.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:08 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:08.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:08 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:10.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:10 smithi023 ceph-mon[31879]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 MiB/s wr, 6.94k op/s 2024-03-16T20:43:10.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:10 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 267 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:10.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:10 smithi031 ceph-mon[36281]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 MiB/s wr, 6.94k op/s 2024-03-16T20:43:10.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:10 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 267 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:10.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:10 smithi189 ceph-mon[36655]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 MiB/s wr, 6.94k op/s 2024-03-16T20:43:10.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:10 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 267 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:12 smithi023 ceph-mon[31879]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2024-03-16T20:43:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:12 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:12.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:12 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:12 smithi031 ceph-mon[36281]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2024-03-16T20:43:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:12 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:12 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:12.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:12 smithi189 ceph-mon[36655]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 6.01k op/s 2024-03-16T20:43:12.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:12 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:12.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:12 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:14.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:14 smithi023 ceph-mon[31879]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 MiB/s wr, 6.50k op/s 2024-03-16T20:43:14.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:14 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 272 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:14.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:14 smithi031 ceph-mon[36281]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 MiB/s wr, 6.50k op/s 2024-03-16T20:43:14.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:14 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 272 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:14.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:14 smithi189 ceph-mon[36655]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 MiB/s wr, 6.50k op/s 2024-03-16T20:43:14.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:14 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 272 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:16.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:16 smithi023 ceph-mon[31879]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 MiB/s wr, 5.39k op/s 2024-03-16T20:43:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:16 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:16 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:16 smithi031 ceph-mon[36281]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 MiB/s wr, 5.39k op/s 2024-03-16T20:43:16.446 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:16 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:16.446 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:16 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:16.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:16 smithi189 ceph-mon[36655]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 MiB/s wr, 5.39k op/s 2024-03-16T20:43:16.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:16 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:16.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:16 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:18.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:18 smithi023 ceph-mon[31879]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-03-16T20:43:18.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:18 smithi031 ceph-mon[36281]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-03-16T20:43:18.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:18 smithi189 ceph-mon[36655]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-03-16T20:43:19.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:19 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 277 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:19.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:19 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 277 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:19.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:19 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 277 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:20 smithi023 ceph-mon[31879]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 5.36k op/s 2024-03-16T20:43:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:20 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:20.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:20 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:20 smithi031 ceph-mon[36281]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 5.36k op/s 2024-03-16T20:43:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:20 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:20.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:20 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:20 smithi189 ceph-mon[36655]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 5.36k op/s 2024-03-16T20:43:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:20 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:20.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:20 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:22.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:22 smithi023 ceph-mon[31879]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 4.47k op/s 2024-03-16T20:43:22.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:22 smithi031 ceph-mon[36281]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 4.47k op/s 2024-03-16T20:43:22.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:22 smithi189 ceph-mon[36655]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 4.47k op/s 2024-03-16T20:43:24.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:24 smithi023 ceph-mon[31879]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 MiB/s wr, 5.27k op/s 2024-03-16T20:43:24.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:24 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 282 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:24.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:24 smithi031 ceph-mon[36281]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 MiB/s wr, 5.27k op/s 2024-03-16T20:43:24.438 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:24 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 282 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:24 smithi189 ceph-mon[36655]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 MiB/s wr, 5.27k op/s 2024-03-16T20:43:24.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:24 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 282 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:25.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:25 smithi023 ceph-mon[31879]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.45k op/s 2024-03-16T20:43:25.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:25 smithi031 ceph-mon[36281]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.45k op/s 2024-03-16T20:43:25.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:25 smithi189 ceph-mon[36655]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.45k op/s 2024-03-16T20:43:27.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:27 smithi189 ceph-mon[36655]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 MiB/s wr, 5.01k op/s 2024-03-16T20:43:27.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:27 smithi023 ceph-mon[31879]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 MiB/s wr, 5.01k op/s 2024-03-16T20:43:27.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:27 smithi031 ceph-mon[36281]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 MiB/s wr, 5.01k op/s 2024-03-16T20:43:28.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:28 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:28.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:28 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:28.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:28 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:28.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:28 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:28.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:28 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:28.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:28 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:29.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:29 smithi189 ceph-mon[36655]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.79k op/s 2024-03-16T20:43:29.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:29 smithi189 ceph-mon[36655]: Health check update: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:29.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:29 smithi023 ceph-mon[31879]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.79k op/s 2024-03-16T20:43:29.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:29 smithi023 ceph-mon[31879]: Health check update: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:29.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:29 smithi031 ceph-mon[36281]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 MiB/s wr, 4.79k op/s 2024-03-16T20:43:29.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:29 smithi031 ceph-mon[36281]: Health check update: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-16T20:43:30.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:30 smithi023 ceph-mon[31879]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops) 2024-03-16T20:43:30.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:30 smithi031 ceph-mon[36281]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops) 2024-03-16T20:43:30.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:30 smithi189 ceph-mon[36655]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 287 sec, osd.9 has slow ops) 2024-03-16T20:43:31.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:31 smithi023 ceph-mon[31879]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 MiB/s wr, 4.91k op/s 2024-03-16T20:43:31.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:31 smithi023 ceph-mon[31879]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 287 secs 2024-03-16T20:43:31.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:31 smithi031 ceph-mon[36281]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 MiB/s wr, 4.91k op/s 2024-03-16T20:43:31.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:31 smithi031 ceph-mon[36281]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 287 secs 2024-03-16T20:43:31.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:31 smithi189 ceph-mon[36655]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 MiB/s wr, 4.91k op/s 2024-03-16T20:43:31.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:31 smithi189 ceph-mon[36655]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 287 secs 2024-03-16T20:43:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:32 smithi023 ceph-mon[31879]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-16T20:43:32.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:32 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:43:32.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:32 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:32.590 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:32 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:32.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:32 smithi031 ceph-mon[36281]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-16T20:43:32.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:32 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:43:32.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:32 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:32.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:32 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:32 smithi189 ceph-mon[36655]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-16T20:43:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:32 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:43:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:32 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/2750734224,v1:172.21.15.23:6839/2750734224] up:standby-replay 2024-03-16T20:43:32.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:32 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:33.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:33 smithi023 ceph-mon[31879]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 4.18k op/s 2024-03-16T20:43:33.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:33 smithi031 ceph-mon[36281]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 4.18k op/s 2024-03-16T20:43:33.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:33 smithi189 ceph-mon[36655]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 4.18k op/s 2024-03-16T20:43:35.522 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:35 smithi189 ceph-mon[36655]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-03-16T20:43:35.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:35 smithi023 ceph-mon[31879]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-03-16T20:43:35.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:35 smithi031 ceph-mon[36281]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-03-16T20:43:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:37 smithi023 ceph-mon[31879]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 1.98k op/s 2024-03-16T20:43:37.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:37 smithi031 ceph-mon[36281]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 1.98k op/s 2024-03-16T20:43:37.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:37 smithi189 ceph-mon[36655]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 1.98k op/s 2024-03-16T20:43:39.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:39 smithi023 ceph-mon[31879]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.4 MiB/s wr, 1.27k op/s 2024-03-16T20:43:39.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:39 smithi031 ceph-mon[36281]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.4 MiB/s wr, 1.27k op/s 2024-03-16T20:43:39.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:39 smithi189 ceph-mon[36655]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.4 MiB/s wr, 1.27k op/s 2024-03-16T20:43:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:41 smithi023 ceph-mon[31879]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.5 MiB/s wr, 370 op/s 2024-03-16T20:43:41.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:41 smithi031 ceph-mon[36281]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.5 MiB/s wr, 370 op/s 2024-03-16T20:43:41.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:41 smithi189 ceph-mon[36655]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.5 MiB/s wr, 370 op/s 2024-03-16T20:43:43.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:43 smithi023 ceph-mon[31879]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-16T20:43:43.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:43 smithi031 ceph-mon[36281]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-16T20:43:43.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:43 smithi189 ceph-mon[36655]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-16T20:43:45.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:45 smithi023 ceph-mon[31879]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-03-16T20:43:45.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:45 smithi031 ceph-mon[36281]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-03-16T20:43:45.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:45 smithi189 ceph-mon[36655]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-03-16T20:43:47.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:47 smithi023 ceph-mon[31879]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:47.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:47 smithi031 ceph-mon[36281]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:47.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:47 smithi189 ceph-mon[36655]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:Syncing()...38 sec 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:FFSB benchmark finished at: Sat Mar 16 20:43:49 2024 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:Results: 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:Benchmark took 356.06 sec 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:Total Results 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout:=============== 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout: ======= ============ ========= ======= =========== ========== 2024-03-16T20:43:49.174 INFO:tasks.workunit.client.0.smithi023.stdout: write : 5313280 14922.55 100.000% 100.000% 58.3MB/sec 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout:- 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout:14922.55 Transactions per Second 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout:Throughput Results 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout:=================== 2024-03-16T20:43:49.175 INFO:tasks.workunit.client.0.smithi023.stdout:Write Throughput: 58.3MB/sec 2024-03-16T20:43:49.198 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.198 INFO:tasks.workunit.client.0.smithi023.stdout:System Call Latency statistics in millisecs 2024-03-16T20:43:49.198 INFO:tasks.workunit.client.0.smithi023.stdout:===== 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: Min Avg Max Total Calls 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: ======== ======== ======== ============ 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout:[ open] 0.004000 0.025546 21.025000 4151 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 1 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 1 2024-03-16T20:43:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 2 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 3 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 8 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 1035 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 2415 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 684 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout:[ write] 0.001000 1.859133 33940.492188 5313280 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 190 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 255 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 199 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 210 2024-03-16T20:43:49.200 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 1404 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 7217 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 4862 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 3257 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 3454 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 1862 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 593 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 434 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 332 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 294 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 265 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 993 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 7859 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 52286 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 5227005 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.201 INFO:tasks.workunit.client.0.smithi023.stdout:[ lseek] 0.000000 0.000656 39.422001 5313280 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 29 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 27 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 15 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 26 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 13 2024-03-16T20:43:49.202 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 15 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 22 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 8 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 49 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 384 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 1112 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 5311580 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout:[ close] 0.001000 0.005073 0.042000 4151 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-16T20:43:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 12 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 19 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 4120 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.204 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.205 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-16T20:43:49.205 INFO:tasks.workunit.client.0.smithi023.stdout:0.7% User Time 2024-03-16T20:43:49.205 INFO:tasks.workunit.client.0.smithi023.stdout:4.8% System Time 2024-03-16T20:43:49.205 INFO:tasks.workunit.client.0.smithi023.stdout:5.4% CPU Utilization 2024-03-16T20:43:49.216 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .. 2024-03-16T20:43:49.217 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -r tmp ffsb 2024-03-16T20:43:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:49 smithi023 ceph-mon[31879]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:49.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:49 smithi031 ceph-mon[36281]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:49.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:49 smithi189 ceph-mon[36655]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-16T20:43:51.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:51 smithi023 ceph-mon[31879]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T20:43:51.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:51 smithi031 ceph-mon[36281]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T20:43:51.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:51 smithi189 ceph-mon[36655]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-16T20:43:52.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:52 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:43:52.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:52 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:43:52.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:52 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:43:53.246 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-16T20:43:53.246 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-16T20:43:53.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:43:53.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:53 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:43:53.582 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-16T20:43:53.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-16T20:43:53.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:43:53.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:53 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:43:53.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:53 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:43:54.520 DEBUG:teuthology.parallel:result is None 2024-03-16T20:43:54.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T20:43:54.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:54 smithi023 ceph-mon[31879]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-16T20:43:54.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:54 smithi023 ceph-mon[31879]: Unable to set osd_memory_target on smithi189 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T20:43:54.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:54 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:54.597 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-16T20:43:54.597 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-16T20:43:54.609 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:43:54.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:54 smithi031 ceph-mon[36281]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-16T20:43:54.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:54 smithi031 ceph-mon[36281]: Unable to set osd_memory_target on smithi189 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T20:43:54.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:54 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:54.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:54 smithi189 ceph-mon[36655]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-16T20:43:54.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:54 smithi189 ceph-mon[36655]: Unable to set osd_memory_target on smithi189 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-16T20:43:54.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:54 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:55.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:55 smithi031 ceph-mon[36281]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 44 op/s 2024-03-16T20:43:55.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:55 smithi189 ceph-mon[36655]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 44 op/s 2024-03-16T20:43:55.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:55 smithi023 ceph-mon[31879]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 44 op/s 2024-03-16T20:43:56.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:56 smithi031 ceph-mon[36281]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:56.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:56 smithi189 ceph-mon[36655]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:56.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:56 smithi023 ceph-mon[31879]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-16T20:43:57.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:57 smithi031 ceph-mon[36281]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 148 KiB/s wr, 51 op/s 2024-03-16T20:43:57.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:57 smithi189 ceph-mon[36655]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 148 KiB/s wr, 51 op/s 2024-03-16T20:43:57.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:57 smithi023 ceph-mon[31879]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 148 KiB/s wr, 51 op/s 2024-03-16T20:43:58.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 -- 172.21.15.23:0/1575314805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 msgr2=0x7f7838099470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:58.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 --2- 172.21.15.23:0/1575314805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f7838099470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f782c009900 tx=0x7f782c02bc60 comp rx=0 tx=0).stop 2024-03-16T20:43:58.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 -- 172.21.15.23:0/1575314805 shutdown_connections 2024-03-16T20:43:58.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 --2- 172.21.15.23:0/1575314805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 0x7f7838099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:58.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 --2- 172.21.15.23:0/1575314805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f7838099470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:58.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 --2- 172.21.15.23:0/1575314805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7838094690 0x7f7838096ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:58.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.351+0000 7f7845089640 1 -- 172.21.15.23:0/1575314805 >> 172.21.15.23:0/1575314805 conn(0x7f7838090290 msgr2=0x7f78380926d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:43:58.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.352+0000 7f7845089640 1 -- 172.21.15.23:0/1575314805 shutdown_connections 2024-03-16T20:43:58.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.352+0000 7f7845089640 1 -- 172.21.15.23:0/1575314805 wait complete. 2024-03-16T20:43:58.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 Processor -- start 2024-03-16T20:43:58.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 -- start start 2024-03-16T20:43:58.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7838094690 0x7f78380a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:58.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:58.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 0x7f78380a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:58.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78380a4300 con 0x7f7838094690 2024-03-16T20:43:58.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78380a4470 con 0x7f78380999b0 2024-03-16T20:43:58.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.524+0000 7f7845089640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78380a4630 con 0x7f7838097000 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:53192/0 (socket says 172.21.15.23:53192) 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 -- 172.21.15.23:0/980436044 learned_addr learned my addr 172.21.15.23:0/980436044 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f7844888640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 0x7f78380a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 -- 172.21.15.23:0/980436044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 msgr2=0x7f78380a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 0x7f78380a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 -- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7838094690 msgr2=0x7f78380a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7838094690 0x7f78380a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.525+0000 7f783f7fe640 1 -- 172.21.15.23:0/980436044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c009560 con 0x7f7838097000 2024-03-16T20:43:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.526+0000 7f783f7fe640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f782c006010 tx=0x7f782c02ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:58.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.528+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c02e2e0 con 0x7f7838097000 2024-03-16T20:43:58.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.528+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78380a48b0 con 0x7f7838097000 2024-03-16T20:43:58.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.528+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78380a4d60 con 0x7f7838097000 2024-03-16T20:43:58.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.528+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f782c02cd20 con 0x7f7838097000 2024-03-16T20:43:58.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.528+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c005420 con 0x7f7838097000 2024-03-16T20:43:58.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.532+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f782c02e440 con 0x7f7838097000 2024-03-16T20:43:58.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.533+0000 7f783d7fa640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 0x7f78300a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:58.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.533+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f782c0ef7b0 con 0x7f7838097000 2024-03-16T20:43:58.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.534+0000 7f783ffff640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 0x7f78300a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:58.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.530+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78240052c0 con 0x7f7838097000 2024-03-16T20:43:58.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.710+0000 7f783ffff640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 0x7f78300a4100 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f783400ba80 tx=0x7f7834005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:58.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:58.870+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f782c0b6e60 con 0x7f7838097000 2024-03-16T20:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.179+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7824005570 con 0x7f7838097000 2024-03-16T20:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.182+0000 7f783d7fa640 1 -- 172.21.15.23:0/980436044 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f782c041030 con 0x7f7838097000 2024-03-16T20:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:43:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 msgr2=0x7f78300a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 0x7f78300a4100 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f783400ba80 tx=0x7f7834005ed0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 msgr2=0x7f78380a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f782c006010 tx=0x7f782c02ed00 comp rx=0 tx=0).stop 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 shutdown_connections 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f78300a1c40 0x7f78300a4100 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78380999b0 0x7f78380a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7838097000 0x7f78380a2d80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 --2- 172.21.15.23:0/980436044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7838094690 0x7f78380a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.185+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 >> 172.21.15.23:0/980436044 conn(0x7f7838090290 msgr2=0x7f7838090670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.186+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 shutdown_connections 2024-03-16T20:43:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.187+0000 7f7845089640 1 -- 172.21.15.23:0/980436044 wait complete. 2024-03-16T20:43:59.275 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:43:59.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 -- 172.21.15.23:0/313724539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc0ffd80 msgr2=0x7fc5cc1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/313724539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc1021b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009900 tx=0x7fc5b802bc60 comp rx=0 tx=0).stop 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 -- 172.21.15.23:0/313724539 shutdown_connections 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/313724539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/313724539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc1026f0 0x7fc5cc104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/313724539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc1021b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 -- 172.21.15.23:0/313724539 >> 172.21.15.23:0/313724539 conn(0x7fc5cc0fb9a0 msgr2=0x7fc5cc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 -- 172.21.15.23:0/313724539 shutdown_connections 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.454+0000 7fc5d2ced640 1 -- 172.21.15.23:0/313724539 wait complete. 2024-03-16T20:43:59.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 Processor -- start 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 -- start start 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc19dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc1026f0 0x7fc5cc19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1a5710 con 0x7fc5cc0ffd80 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1a5880 con 0x7fc5cc1026f0 2024-03-16T20:43:59.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.455+0000 7fc5d2ced640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1a5a40 con 0x7fc5cc1050a0 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:53224/0 (socket says 172.21.15.23:53224) 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 -- 172.21.15.23:0/1322802567 learned_addr learned my addr 172.21.15.23:0/1322802567 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d0a89640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc19dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5cbfff640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc1026f0 0x7fc5cc19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 -- 172.21.15.23:0/1322802567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc1026f0 msgr2=0x7fc5cc19e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc1026f0 0x7fc5cc19e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 -- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 msgr2=0x7fc5cc19dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc19dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d128a640 1 -- 172.21.15.23:0/1322802567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8009560 con 0x7fc5cc1050a0 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.456+0000 7fc5d0a89640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc19dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.457+0000 7fc5d128a640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400b990 tx=0x7fc5c400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:59.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.459+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400cba0 con 0x7fc5cc1050a0 2024-03-16T20:43:59.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.459+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5c400cd00 con 0x7fc5cc1050a0 2024-03-16T20:43:59.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.459+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4007750 con 0x7fc5cc1050a0 2024-03-16T20:43:59.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.459+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc1a5d20 con 0x7fc5cc1050a0 2024-03-16T20:43:59.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.459+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc1a6150 con 0x7fc5cc1050a0 2024-03-16T20:43:59.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.461+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc0fd490 con 0x7fc5cc1050a0 2024-03-16T20:43:59.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.462+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc5c4014a40 con 0x7fc5cc1050a0 2024-03-16T20:43:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.463+0000 7fc5c9ffb640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 0x7fc5bc0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.463+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fc5c40cd090 con 0x7fc5cc1050a0 2024-03-16T20:43:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.463+0000 7fc5d0a89640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 0x7fc5bc0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.464+0000 7fc5d0a89640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 0x7fc5bc0a4040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8000c00 tx=0x7fc5b802c810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:59.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.472+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc5c4095f20 con 0x7fc5cc1050a0 2024-03-16T20:43:59.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:59 smithi023 ceph-mon[31879]: pgmap v596: 129 pgs: 129 active+clean; 8.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 153 KiB/s wr, 120 op/s 2024-03-16T20:43:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:43:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/980436044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:43:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:59 smithi031 ceph-mon[36281]: pgmap v596: 129 pgs: 129 active+clean; 8.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 153 KiB/s wr, 120 op/s 2024-03-16T20:43:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:43:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/980436044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:43:59.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:59 smithi189 ceph-mon[36655]: pgmap v596: 129 pgs: 129 active+clean; 8.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 153 KiB/s wr, 120 op/s 2024-03-16T20:43:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:43:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/980436044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.740+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5cc02d000 con 0x7fc5cc1050a0 2024-03-16T20:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.740+0000 7fc5c9ffb640 1 -- 172.21.15.23:0/1322802567 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5c40190a0 con 0x7fc5cc1050a0 2024-03-16T20:43:59.743 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 msgr2=0x7fc5bc0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 0x7fc5bc0a4040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8000c00 tx=0x7fc5b802c810 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 msgr2=0x7fc5cc1a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400b990 tx=0x7fc5c400be50 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 shutdown_connections 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc5bc0a1b80 0x7fc5bc0a4040 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5cc1050a0 0x7fc5cc1a51a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5cc1026f0 0x7fc5cc19e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 --2- 172.21.15.23:0/1322802567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5cc0ffd80 0x7fc5cc19dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 >> 172.21.15.23:0/1322802567 conn(0x7fc5cc0fb9a0 msgr2=0x7fc5cc103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 shutdown_connections 2024-03-16T20:43:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.742+0000 7fc5d2ced640 1 -- 172.21.15.23:0/1322802567 wait complete. 2024-03-16T20:43:59.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:43:59.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 -- 172.21.15.23:0/1740807873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec096820 msgr2=0x7fd9ec09d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1740807873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec096820 0x7fd9ec09d0b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd9e401a910 tx=0x7fd9e403cc60 comp rx=0 tx=0).stop 2024-03-16T20:43:59.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 -- 172.21.15.23:0/1740807873 shutdown_connections 2024-03-16T20:43:59.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1740807873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec096820 0x7fd9ec09d0b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1740807873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec0962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1740807873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec094c70 0x7fd9ec095090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.921+0000 7fd9f359e640 1 -- 172.21.15.23:0/1740807873 >> 172.21.15.23:0/1740807873 conn(0x7fd9ec0901c0 msgr2=0x7fd9ec092620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:43:59.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.922+0000 7fd9f359e640 1 -- 172.21.15.23:0/1740807873 shutdown_connections 2024-03-16T20:43:59.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.922+0000 7fd9f359e640 1 -- 172.21.15.23:0/1740807873 wait complete. 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.922+0000 7fd9f359e640 1 Processor -- start 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 -- start start 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 0x7fd9ec12df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec12e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec096820 0x7fd9ec135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec135a40 con 0x7fd9ec094c70 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec135bb0 con 0x7fd9ec095e60 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f359e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec135d70 con 0x7fd9ec096820 2024-03-16T20:43:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 0x7fd9ec12df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 0x7fd9ec12df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39542/0 (socket says 172.21.15.23:39542) 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec12e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f259c640 1 -- 172.21.15.23:0/1054413444 learned_addr learned my addr 172.21.15.23:0/1054413444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f2d9d640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec096820 0x7fd9ec135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 -- 172.21.15.23:0/1054413444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec096820 msgr2=0x7fd9ec135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec096820 0x7fd9ec135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 -- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 msgr2=0x7fd9ec12df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 0x7fd9ec12df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.923+0000 7fd9f1d9b640 1 -- 172.21.15.23:0/1054413444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e401a570 con 0x7fd9ec095e60 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9f1d9b640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec12e490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8005fd0 tx=0x7fd9e800dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8009c30 con 0x7fd9ec095e60 2024-03-16T20:43:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec135ff0 con 0x7fd9ec095e60 2024-03-16T20:43:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec136520 con 0x7fd9ec095e60 2024-03-16T20:43:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e8004510 con 0x7fd9ec095e60 2024-03-16T20:43:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.924+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8003cc0 con 0x7fd9ec095e60 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.925+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d40052c0 con 0x7fd9ec095e60 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.927+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd9e8003e20 con 0x7fd9ec095e60 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.927+0000 7fd9e37fe640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 0x7fd9dc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.927+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fd9e80ccd50 con 0x7fd9ec095e60 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.927+0000 7fd9f259c640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 0x7fd9dc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:43:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.928+0000 7fd9f259c640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 0x7fd9dc0a3ef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40504c0 tx=0x7fd9f4071040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:43:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:43:59.934+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd9e8095c20 con 0x7fd9ec095e60 2024-03-16T20:44:00.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.258+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9d4005af0 con 0x7fd9ec095e60 2024-03-16T20:44:00.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.261+0000 7fd9e37fe640 1 -- 172.21.15.23:0/1054413444 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9e8020020 con 0x7fd9ec095e60 2024-03-16T20:44:00.265 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.265+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 msgr2=0x7fd9dc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 0x7fd9dc0a3ef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40504c0 tx=0x7fd9f4071040 comp rx=0 tx=0).stop 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 msgr2=0x7fd9ec12e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec12e490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8005fd0 tx=0x7fd9e800dd20 comp rx=0 tx=0).stop 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 shutdown_connections 2024-03-16T20:44:00.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd9dc0a1a30 0x7fd9dc0a3ef0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9ec096820 0x7fd9ec135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9ec095e60 0x7fd9ec12e490 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 --2- 172.21.15.23:0/1054413444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec094c70 0x7fd9ec12df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 >> 172.21.15.23:0/1054413444 conn(0x7fd9ec0901c0 msgr2=0x7fd9ec099100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 shutdown_connections 2024-03-16T20:44:00.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.266+0000 7fd9f359e640 1 -- 172.21.15.23:0/1054413444 wait complete. 2024-03-16T20:44:00.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:44:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.446+0000 7fedc1bd7640 1 -- 172.21.15.23:0/602500815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc100ed0 msgr2=0x7fedbc105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.446+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/602500815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc100ed0 0x7fedbc105770 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009ca0 tx=0x7fedb402c120 comp rx=0 tx=0).stop 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 -- 172.21.15.23:0/602500815 shutdown_connections 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/602500815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc100ed0 0x7fedbc105770 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/602500815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedbc100510 0x7fedbc100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/602500815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 -- 172.21.15.23:0/602500815 >> 172.21.15.23:0/602500815 conn(0x7fedbc0fb0e0 msgr2=0x7fedbc0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 -- 172.21.15.23:0/602500815 shutdown_connections 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.451+0000 7fedc1bd7640 1 -- 172.21.15.23:0/602500815 wait complete. 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 Processor -- start 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 -- start start 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedbc100ed0 0x7fedbc072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc10cc60 con 0x7fedbc0ff7b0 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc10cdd0 con 0x7fedbc100ed0 2024-03-16T20:44:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.452+0000 7fedc1bd7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc10cf90 con 0x7fedbc100510 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:53268/0 (socket says 172.21.15.23:53268) 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 -- 172.21.15.23:0/3474576843 learned_addr learned my addr 172.21.15.23:0/3474576843 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbb7fe640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 -- 172.21.15.23:0/3474576843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedbc100ed0 msgr2=0x7fedbc072540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedbc100ed0 0x7fedbc072540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 -- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 msgr2=0x7fedbc071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc071ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbaffd640 1 -- 172.21.15.23:0/3474576843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac009490 con 0x7fedbc100510 2024-03-16T20:44:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.453+0000 7fedbb7fe640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:44:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.454+0000 7fedbaffd640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c970 tx=0x7fedb000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.456+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0007c10 con 0x7fedbc100510 2024-03-16T20:44:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.456+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb0007d70 con 0x7fedbc100510 2024-03-16T20:44:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.456+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb000e740 con 0x7fedbc100510 2024-03-16T20:44:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.457+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb4009900 con 0x7fedbc100510 2024-03-16T20:44:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.457+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc10d5a0 con 0x7fedbc100510 2024-03-16T20:44:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.460+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc113570 con 0x7fedbc100510 2024-03-16T20:44:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.460+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fedb000fa30 con 0x7fedbc100510 2024-03-16T20:44:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.461+0000 7fedb8ff9640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 0x7feda40a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.461+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fedb0013070 con 0x7fedbc100510 2024-03-16T20:44:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.463+0000 7fedbb7fe640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 0x7feda40a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.464+0000 7fedbb7fe640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 0x7feda40a40f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fedac009ce0 tx=0x7fedac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:00.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.471+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fedb0096a30 con 0x7fedbc100510 2024-03-16T20:44:00.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1322802567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/1054413444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1322802567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/1054413444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1322802567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/1054413444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:00.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.816+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fedbc113700 con 0x7fedbc100510 2024-03-16T20:44:00.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.817+0000 7fedb8ff9640 1 -- 172.21.15.23:0/3474576843 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fedb009a0f0 con 0x7fedbc100510 2024-03-16T20:44:00.820 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:44:00.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 msgr2=0x7feda40a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 0x7feda40a40f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fedac009ce0 tx=0x7fedac005f90 comp rx=0 tx=0).stop 2024-03-16T20:44:00.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 msgr2=0x7fedbc072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:00.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fedb000c970 tx=0x7fedb000ce30 comp rx=0 tx=0).stop 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 shutdown_connections 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7feda40a1c30 0x7feda40a40f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedbc100ed0 0x7fedbc072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fedbc100510 0x7fedbc072000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 --2- 172.21.15.23:0/3474576843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedbc0ff7b0 0x7fedbc071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:00.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 >> 172.21.15.23:0/3474576843 conn(0x7fedbc0fb0e0 msgr2=0x7fedbc103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:00.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.820+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 shutdown_connections 2024-03-16T20:44:00.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.823+0000 7fedc1bd7640 1 -- 172.21.15.23:0/3474576843 wait complete. 2024-03-16T20:44:00.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 -- 172.21.15.23:0/2326563124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe914096f50 msgr2=0x7fe9140993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 --2- 172.21.15.23:0/2326563124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe914096f50 0x7fe9140993c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe91001a910 tx=0x7fe91003cca0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 -- 172.21.15.23:0/2326563124 shutdown_connections 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 --2- 172.21.15.23:0/2326563124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe914099900 0x7fe914099d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 --2- 172.21.15.23:0/2326563124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe914096f50 0x7fe9140993c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 --2- 172.21.15.23:0/2326563124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9140945e0 0x7fe914096a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.998+0000 7fe91bfff640 1 -- 172.21.15.23:0/2326563124 >> 172.21.15.23:0/2326563124 conn(0x7fe9140901c0 msgr2=0x7fe914092620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.999+0000 7fe91bfff640 1 -- 172.21.15.23:0/2326563124 shutdown_connections 2024-03-16T20:44:01.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:00.999+0000 7fe91bfff640 1 -- 172.21.15.23:0/2326563124 wait complete. 2024-03-16T20:44:01.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 Processor -- start 2024-03-16T20:44:01.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 -- start start 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe914096f50 0x7fe91413b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 0x7fe9141422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914142820 con 0x7fe9140945e0 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914142990 con 0x7fe914099900 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91bfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914142b50 con 0x7fe914096f50 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.000+0000 7fe91affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56436/0 (socket says 172.21.15.23:56436) 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 -- 172.21.15.23:0/3019130941 learned_addr learned my addr 172.21.15.23:0/3019130941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 -- 172.21.15.23:0/3019130941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe914096f50 msgr2=0x7fe91413b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe914096f50 0x7fe91413b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 -- 172.21.15.23:0/3019130941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 msgr2=0x7fe9141422b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91b7fe640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 0x7fe9141422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.001+0000 7fe91affd640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 0x7fe9141422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe91affd640 1 -- 172.21.15.23:0/3019130941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91001a570 con 0x7fe9140945e0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe91b7fe640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 0x7fe9141422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe91affd640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b450 tx=0x7fe90c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c011890 con 0x7fe9140945e0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe90c0103e0 con 0x7fe9140945e0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c00f5a0 con 0x7fe9140945e0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe914142e30 con 0x7fe9140945e0 2024-03-16T20:44:01.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.002+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe914143330 con 0x7fe9140945e0 2024-03-16T20:44:01.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.005+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91409f160 con 0x7fe9140945e0 2024-03-16T20:44:01.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.006+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe90c010550 con 0x7fe9140945e0 2024-03-16T20:44:01.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.007+0000 7fe903fff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 0x7fe9040a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.007+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fe90c0172c0 con 0x7fe9140945e0 2024-03-16T20:44:01.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.010+0000 7fe91a7fc640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 0x7fe9040a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.014+0000 7fe91a7fc640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 0x7fe9040a3fc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe910004770 tx=0x7fe910000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:01.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.017+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe90c096bf0 con 0x7fe9140945e0 2024-03-16T20:44:01.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.360+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe914143610 con 0x7fe9140945e0 2024-03-16T20:44:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.362+0000 7fe903fff640 1 -- 172.21.15.23:0/3019130941 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe90c0150a0 con 0x7fe9140945e0 2024-03-16T20:44:01.365 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:44:01.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 msgr2=0x7fe9040a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 0x7fe9040a3fc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe910004770 tx=0x7fe910000bc0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 msgr2=0x7fe91413b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b450 tx=0x7fe90c00b910 comp rx=0 tx=0).stop 2024-03-16T20:44:01.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 shutdown_connections 2024-03-16T20:44:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe9040a1b00 0x7fe9040a3fc0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe910004770 tx=0x7fe910000bc0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe914099900 0x7fe9141422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe914096f50 0x7fe91413b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 --2- 172.21.15.23:0/3019130941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9140945e0 0x7fe91413b360 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b450 tx=0x7fe90c00b910 comp rx=0 tx=0).stop 2024-03-16T20:44:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.365+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 >> 172.21.15.23:0/3019130941 conn(0x7fe9140901c0 msgr2=0x7fe9140925b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:01.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.367+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 shutdown_connections 2024-03-16T20:44:01.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.368+0000 7fe91bfff640 1 -- 172.21.15.23:0/3019130941 wait complete. 2024-03-16T20:44:01.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-16T20:44:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.571+0000 7f7964148640 1 -- 172.21.15.23:0/3705536494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10ae60 msgr2=0x7f795c10d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.571+0000 7f7964148640 1 --2- 172.21.15.23:0/3705536494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10ae60 0x7f795c10d2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802bc80 comp rx=0 tx=0).stop 2024-03-16T20:44:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.573+0000 7f7964148640 1 -- 172.21.15.23:0/3705536494 shutdown_connections 2024-03-16T20:44:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.573+0000 7f7964148640 1 --2- 172.21.15.23:0/3705536494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10d830 0x7f795c10dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.573+0000 7f7964148640 1 --2- 172.21.15.23:0/3705536494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10ae60 0x7f795c10d2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.573+0000 7f7964148640 1 --2- 172.21.15.23:0/3705536494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c10a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.573+0000 7f7964148640 1 -- 172.21.15.23:0/3705536494 >> 172.21.15.23:0/3705536494 conn(0x7f795c06e360 msgr2=0x7f795c0707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:01.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.574+0000 7f7964148640 1 -- 172.21.15.23:0/3705536494 shutdown_connections 2024-03-16T20:44:01.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.574+0000 7f7964148640 1 -- 172.21.15.23:0/3705536494 wait complete. 2024-03-16T20:44:01.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.574+0000 7f7964148640 1 Processor -- start 2024-03-16T20:44:01.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.575+0000 7f7964148640 1 -- start start 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.575+0000 7f7964148640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c116a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.575+0000 7f7964148640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 0x7f795c116fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.575+0000 7f7964148640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 0x7f795c1b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.575+0000 7f7964148640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c11a020 con 0x7f795c1084f0 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f7964148640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c11a190 con 0x7f795c10ae60 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f7964148640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c11a350 con 0x7f795c10d830 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f79626e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 0x7f795c1b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f79626e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 0x7f795c1b65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.23:45660/0 (socket says 172.21.15.23:45660) 2024-03-16T20:44:01.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f79626e5640 1 -- 172.21.15.23:0/306371329 learned_addr learned my addr 172.21.15.23:0/306371329 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:01.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.576+0000 7f7961ee4640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c116a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f79616e3640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 0x7f795c116fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f7961ee4640 1 -- 172.21.15.23:0/306371329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 msgr2=0x7f795c1b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f7961ee4640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 0x7f795c1b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f7961ee4640 1 -- 172.21.15.23:0/306371329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 msgr2=0x7f795c116fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f7961ee4640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 0x7f795c116fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f7961ee4640 1 -- 172.21.15.23:0/306371329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958009560 con 0x7f795c1084f0 2024-03-16T20:44:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.577+0000 7f79616e3640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 0x7f795c116fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:44:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.578+0000 7f7961ee4640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c116a90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f794c00d6a0 tx=0x7f794c00db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.578+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c004220 con 0x7f795c1084f0 2024-03-16T20:44:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.578+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c1b6cc0 con 0x7f795c1084f0 2024-03-16T20:44:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.578+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c1b71d0 con 0x7f795c1084f0 2024-03-16T20:44:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.578+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f794c004380 con 0x7f795c1084f0 2024-03-16T20:44:01.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.579+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c003d80 con 0x7f795c1084f0 2024-03-16T20:44:01.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.581+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79400052c0 con 0x7f795c1084f0 2024-03-16T20:44:01.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.582+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f794c009820 con 0x7f795c1084f0 2024-03-16T20:44:01.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.583+0000 7f7952ffd640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 0x7f79480a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:01.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.583+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f794c0cd090 con 0x7f795c1084f0 2024-03-16T20:44:01.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.585+0000 7f79616e3640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 0x7f79480a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:01.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.587+0000 7f79616e3640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 0x7f79480a3f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f795c117fa0 tx=0x7f7958005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.591+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f794c095f00 con 0x7f795c1084f0 2024-03-16T20:44:01.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:01 smithi031 ceph-mon[36281]: pgmap v597: 129 pgs: 129 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 150 op/s 2024-03-16T20:44:01.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3474576843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3019130941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:01 smithi189 ceph-mon[36655]: pgmap v597: 129 pgs: 129 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 150 op/s 2024-03-16T20:44:01.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3474576843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3019130941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:01 smithi023 ceph-mon[31879]: pgmap v597: 129 pgs: 129 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 150 op/s 2024-03-16T20:44:01.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3474576843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3019130941' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:01.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.987+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f7940005570 con 0x7f795c1084f0 2024-03-16T20:44:02.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:01.999+0000 7f7952ffd640 1 -- 172.21.15.23:0/306371329 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+734469 (secure 0 0 0) 0x7f794c020090 con 0x7f795c1084f0 2024-03-16T20:44:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:44:02.013 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:44:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 20, 2024-03-16T20:44:02.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 14152, 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3040096369 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-16T20:44:02.014 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3040096369 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/3040096369", 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2024-03-16T20:26:49.857080+0000", 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540701547738038271, 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 24109, 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-16T20:44:02.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.018 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.020 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.030 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.031 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T20:44:02.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.041 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-16T20:44:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.049 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.051 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.052 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-16T20:44:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-16T20:44:02.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-16T20:44:02.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-16T20:44:02.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T20:44:02.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T20:44:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.073 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.074 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T20:44:02.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-16T20:44:02.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-16T20:44:02.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.083 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.087 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-16T20:44:02.094 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.101 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-16T20:44:02.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.108 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.109 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.111 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.115 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.121 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.125 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T20:44:02.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T20:44:02.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T20:44:02.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.129 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T20:44:02.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-16T20:44:02.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T20:44:02.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T20:44:02.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T20:44:02.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T20:44:02.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.150 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.161 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.162 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.164 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:02.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-16T20:44:02.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:02.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.174 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.177 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.181 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-16T20:44:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-16T20:44:02.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-16T20:44:02.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-16T20:44:02.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-16T20:44:02.188 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.189 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.192 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-16T20:44:02.196 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.201 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-16T20:44:02.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-16T20:44:02.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.216 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.220 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T20:44:02.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.223 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.229 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-16T20:44:02.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.238 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.239 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.243 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.252 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:02.254 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T20:44:02.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.256 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.259 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-16T20:44:02.261 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.262 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.263 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.266 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.268 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.270 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.276 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.285 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.286 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-16T20:44:02.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-16T20:44:02.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.292 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.297 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.299 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-16T20:44:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:02.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.311 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.321 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.322 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.324 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T20:44:02.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.333 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.335 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.343 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.352 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.354 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.356 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.357 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.359 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.362 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.364 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.366 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.371 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-16T20:44:02.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.380 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-16T20:44:02.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.385 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.386 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.395 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.401 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-16T20:44:02.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.407 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.409 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-16T20:44:02.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T20:44:02.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 24119, 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-16T20:44:02.416 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T20:44:02.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.418 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.421 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.425 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-16T20:44:02.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.430 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-16T20:44:02.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-16T20:44:02.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-16T20:44:02.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-16T20:44:02.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-16T20:44:02.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-16T20:44:02.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.441 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-16T20:44:02.444 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T20:44:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.448 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-16T20:44:02.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.453 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.455 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.460 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.462 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-16T20:44:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-16T20:44:02.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-16T20:44:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T20:44:02.471 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-16T20:44:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-16T20:44:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-16T20:44:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.485 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.486 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.493 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.495 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.497 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-16T20:44:02.498 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-16T20:44:02.499 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-16T20:44:02.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.503 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.505 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.507 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.509 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-16T20:44:02.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-16T20:44:02.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T20:44:02.514 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T20:44:02.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T20:44:02.516 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.518 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.520 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.521 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-16T20:44:02.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.526 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.530 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T20:44:02.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T20:44:02.532 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.533 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.534 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T20:44:02.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T20:44:02.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.537 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.539 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.542 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.543 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.549 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.550 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.551 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T20:44:02.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T20:44:02.553 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T20:44:02.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T20:44:02.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T20:44:02.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.560 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.562 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-16T20:44:02.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.571 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.573 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-16T20:44:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.602 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-16T20:44:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-16T20:44:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-16T20:44:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-16T20:44:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.616 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T20:44:02.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.623 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.625 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.628 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.629 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.631 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.633 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-16T20:44:02.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.635 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-16T20:44:02.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.644 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-16T20:44:02.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.650 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.654 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.655 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.658 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.661 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.662 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.668 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.671 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.672 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.675 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.678 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.679 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.680 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.681 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.682 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.683 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.689 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.690 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:02.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-16T20:44:02.694 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.707 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-16T20:44:02.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:02.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-16T20:44:02.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-16T20:44:02.717 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:02.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.721 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.724 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T20:44:02.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.729 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.734 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.735 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:02.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.739 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.746 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-16T20:44:02.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.764 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.765 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.767 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.772 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.773 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.774 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.775 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.776 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.784 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.789 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.790 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.791 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.793 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.795 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-16T20:44:02.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-16T20:44:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-16T20:44:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-16T20:44:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-16T20:44:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.836 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-16T20:44:02.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.838 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "restful" 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-16T20:44:02.839 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.840 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.843 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.844 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-16T20:44:02.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.850 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.852 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-16T20:44:02.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-16T20:44:02.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-16T20:44:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-16T20:44:02.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-16T20:44:02.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.862 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.864 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.867 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:02.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.873 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.875 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.876 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.877 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-16T20:44:02.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-16T20:44:02.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-16T20:44:02.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-16T20:44:02.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-16T20:44:02.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-16T20:44:02.886 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-16T20:44:02.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-16T20:44:02.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-16T20:44:02.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-16T20:44:02.890 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-16T20:44:02.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-16T20:44:02.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-16T20:44:02.894 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.897 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.898 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.899 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-16T20:44:02.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.902 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-16T20:44:02.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.910 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.912 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.914 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.916 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.918 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-16T20:44:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-16T20:44:02.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.921 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-16T20:44:02.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-16T20:44:02.923 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.924 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.928 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.930 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-16T20:44:02.933 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-16T20:44:02.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-16T20:44:02.935 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.937 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.941 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.946 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:02.947 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-16T20:44:02.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.950 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-16T20:44:02.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.966 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.970 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.984 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-16T20:44:02.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.986 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:02.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.990 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.994 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.995 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.997 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:02.998 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:02.999 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.000 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.001 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.007 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:03.008 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.012 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.015 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.016 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-16T20:44:03.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.024 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-16T20:44:03.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-16T20:44:03.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-16T20:44:03.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.043 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.044 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-16T20:44:03.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.047 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.051 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.054 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.055 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.057 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.060 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-16T20:44:03.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.066 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-16T20:44:03.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.075 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.076 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.081 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.085 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.096 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.100 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.101 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.102 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.103 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.109 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.110 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.112 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-16T20:44:03.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-16T20:44:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.121 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.127 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-16T20:44:03.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-16T20:44:03.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.148 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.150 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.152 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.156 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.157 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.160 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.161 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.165 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.169 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.172 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-16T20:44:03.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-16T20:44:03.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-16T20:44:03.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.183 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.189 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.195 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.200 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.202 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-16T20:44:03.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:03.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.207 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.209 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-16T20:44:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-16T20:44:03.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.214 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-16T20:44:03.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.227 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.233 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.239 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.241 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-16T20:44:03.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-16T20:44:03.246 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.248 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.249 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.250 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.251 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-16T20:44:03.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-16T20:44:03.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.257 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2024-03-16T20:44:03.258 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-16T20:44:03.259 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-16T20:44:03.260 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "squid": [ 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-16T20:44:03.261 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 3, 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3105884864 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.262 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3890971788 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-16T20:44:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 885364000 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1087781272 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:03.265 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-16T20:44:03.265 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:44:03.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/306371329' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T20:44:03.266 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/306371329' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 msgr2=0x7f79480a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 0x7f79480a3f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f795c117fa0 tx=0x7f7958005dc0 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 msgr2=0x7f795c116a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c116a90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f794c00d6a0 tx=0x7f794c00db60 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 shutdown_connections 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f79480a1a60 0x7f79480a3f20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c10d830 0x7f795c1b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795c10ae60 0x7f795c116fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 --2- 172.21.15.23:0/306371329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f795c1084f0 0x7f795c116a90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 >> 172.21.15.23:0/306371329 conn(0x7f795c06e360 msgr2=0x7f795c10b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 shutdown_connections 2024-03-16T20:44:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:02.007+0000 7f7964148640 1 -- 172.21.15.23:0/306371329 wait complete. 2024-03-16T20:44:03.273 DEBUG:tasks.check_counter:Getting stats from a 2024-03-16T20:44:03.273 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-16T20:44:03.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/306371329' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-16T20:44:03.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:03 smithi023 ceph-mon[31879]: pgmap v598: 129 pgs: 129 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 160 KiB/s wr, 193 op/s 2024-03-16T20:44:03.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:03 smithi031 ceph-mon[36281]: pgmap v598: 129 pgs: 129 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 160 KiB/s wr, 193 op/s 2024-03-16T20:44:03.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:03 smithi189 ceph-mon[36655]: pgmap v598: 129 pgs: 129 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 160 KiB/s wr, 193 op/s 2024-03-16T20:44:05.342 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:44:05.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:05 smithi031 ceph-mon[36281]: pgmap v599: 129 pgs: 129 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 217 op/s 2024-03-16T20:44:05.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:05 smithi189 ceph-mon[36655]: pgmap v599: 129 pgs: 129 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 217 op/s 2024-03-16T20:44:05.838 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:05 smithi023 ceph-mon[31879]: pgmap v599: 129 pgs: 129 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 217 op/s 2024-03-16T20:44:07.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:07 smithi031 ceph-mon[36281]: pgmap v600: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-03-16T20:44:07.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:07 smithi189 ceph-mon[36655]: pgmap v600: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-03-16T20:44:07.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:07 smithi023 ceph-mon[31879]: pgmap v600: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-03-16T20:44:10.141 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:10 smithi023 ceph-mon[31879]: pgmap v601: 129 pgs: 129 active+clean; 504 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 253 op/s 2024-03-16T20:44:10.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:10 smithi031 ceph-mon[36281]: pgmap v601: 129 pgs: 129 active+clean; 504 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 253 op/s 2024-03-16T20:44:10.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:10 smithi189 ceph-mon[36655]: pgmap v601: 129 pgs: 129 active+clean; 504 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 KiB/s wr, 253 op/s 2024-03-16T20:44:12.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:12 smithi023 ceph-mon[31879]: pgmap v602: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 KiB/s wr, 201 op/s 2024-03-16T20:44:12.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:12 smithi031 ceph-mon[36281]: pgmap v602: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 KiB/s wr, 201 op/s 2024-03-16T20:44:12.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:12 smithi189 ceph-mon[36655]: pgmap v602: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 KiB/s wr, 201 op/s 2024-03-16T20:44:14.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:14 smithi023 ceph-mon[31879]: pgmap v603: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 172 op/s 2024-03-16T20:44:14.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:14 smithi031 ceph-mon[36281]: pgmap v603: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 172 op/s 2024-03-16T20:44:14.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:14 smithi189 ceph-mon[36655]: pgmap v603: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 172 op/s 2024-03-16T20:44:16.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:16 smithi023 ceph-mon[31879]: pgmap v604: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 129 op/s 2024-03-16T20:44:16.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:16 smithi031 ceph-mon[36281]: pgmap v604: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 129 op/s 2024-03-16T20:44:16.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:16 smithi189 ceph-mon[36655]: pgmap v604: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 129 op/s 2024-03-16T20:44:18.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:18 smithi023 ceph-mon[31879]: pgmap v605: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 96 op/s 2024-03-16T20:44:18.357 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:44:18.357 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1169, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 1169, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 352605, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 362764, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.102533557, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.031232105, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.027349303, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.035411261, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1168, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.060122043, 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000051474 2024-03-16T20:44:18.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 469, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 599, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 115940, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 450368, 2024-03-16T20:44:18.359 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 4, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.057546890, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.018075610, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.010911418, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.020634404, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 597, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.029816839, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000049944 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 304752 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 217, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 181, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 950300, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 104727, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.030762030, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.010968282, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.019992326, 2024-03-16T20:44:18.361 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003807848, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 180, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.012958464, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000071991 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 950032, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 104416 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-16T20:44:18.362 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2745, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.083036817, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000030250 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.363 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-16T20:44:18.364 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 1, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:18.365 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:18.366 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-16T20:44:18.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:18.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 5, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-16T20:44:18.369 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 2, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 2, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 4195732, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 4195424, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 5 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 3, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 3, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 1, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 1, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-16T20:44:18.371 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 52644, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 215420 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.372 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.374 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.375 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:18.380 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-16T20:44:18.381 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:18.382 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 1, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 2, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 1, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 1, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-16T20:44:18.383 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 1598, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:18.384 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:18.385 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 189463, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 52, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-16T20:44:18.386 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 164, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 8320, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 13, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-16T20:44:18.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 1634, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 1634, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1634, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 27.845776430, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.017041478 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1634, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1635 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 1634, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 1634, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1045, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 587, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-16T20:44:18.389 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-16T20:44:18.390 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-16T20:44:18.391 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 92, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 2, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 5, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 5, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-16T20:44:18.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:18.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 1851, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1238140, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 1851, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 1851, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1238140, 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:18.394 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 1634, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 1433, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 1634, 2024-03-16T20:44:18.395 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1433, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 1634, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 1634, 2024-03-16T20:44:18.396 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 1634, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1634, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:18.397 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x559585f26960": { 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:18.398 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:18.399 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:44:18.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:18 smithi031 ceph-mon[36281]: pgmap v605: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 96 op/s 2024-03-16T20:44:18.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:18 smithi189 ceph-mon[36655]: pgmap v605: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 96 op/s 2024-03-16T20:44:19.074 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-16T20:44:19.074 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-16T20:44:19.074 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-16T20:44:19.074 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-16T20:44:19.074 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-16T20:44:19.074 DEBUG:tasks.check_counter:Getting stats from d 2024-03-16T20:44:19.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-16T20:44:19.331 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:44:20.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:20 smithi031 ceph-mon[36281]: pgmap v606: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 85 op/s 2024-03-16T20:44:20.452 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:20 smithi023 ceph-mon[31879]: pgmap v606: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 85 op/s 2024-03-16T20:44:20.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:20 smithi189 ceph-mon[36655]: pgmap v606: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 85 op/s 2024-03-16T20:44:21.111 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:44:21.111 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001094512, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000189188, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:21.112 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 135, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 101152, 2024-03-16T20:44:21.113 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.010397535, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007939558, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001245386, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 133, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.009913738, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000074539 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:21.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 101152 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 187, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 180, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 738560, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 106800, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.031514265, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.011187991, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.017831426, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003611013, 2024-03-16T20:44:21.115 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 180, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.011603077, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000064461 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 738560, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 106800 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-16T20:44:21.116 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:21.117 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:21.118 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149007, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 30, 2024-03-16T20:44:21.119 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-16T20:44:21.120 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 184, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 709654, 2024-03-16T20:44:21.121 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 184, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 184, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 709654, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:44:21.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:21 smithi023 ceph-mon[31879]: pgmap v607: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 26 op/s 2024-03-16T20:44:21.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:21 smithi031 ceph-mon[36281]: pgmap v607: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 26 op/s 2024-03-16T20:44:21.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:21 smithi189 ceph-mon[36655]: pgmap v607: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 26 op/s 2024-03-16T20:44:21.788 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-16T20:44:21.788 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-16T20:44:21.788 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-16T20:44:21.788 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-16T20:44:21.788 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-16T20:44:21.788 DEBUG:tasks.check_counter:Getting stats from g 2024-03-16T20:44:21.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-16T20:44:22.043 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:44:23.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:23 smithi189 ceph-mon[36655]: pgmap v608: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T20:44:23.481 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:23 smithi023 ceph-mon[31879]: pgmap v608: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T20:44:23.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:23 smithi031 ceph-mon[36281]: pgmap v608: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 296, 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 354, 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 101129, 2024-03-16T20:44:23.733 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 95727, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.033807883, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.014674285, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.007269215, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.007998456, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 353, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.022535005, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000063838 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.734 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 694, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 658, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1123657, 2024-03-16T20:44:23.735 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 255997, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.061824049, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.019640076, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.027396952, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.016316822, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 657, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.023724589, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000036110 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:23.736 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 950576, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 107904 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 135, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1508, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 305872, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-16T20:44:23.737 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 4, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.009723431, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007002409, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001684608, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000071120, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 133, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.013157197, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000098926 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:23.738 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 304928 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-16T20:44:23.739 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1351, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.031991870, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000023680 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.740 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-16T20:44:23.741 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 4, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 4, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 4, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 1, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-16T20:44:23.742 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-16T20:44:23.743 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:23.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:23.746 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 27, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 2, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 2, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-16T20:44:23.747 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 4245569, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 4243918, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 27 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 4, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 20, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 16, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 2, 2024-03-16T20:44:23.748 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 16, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 14, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 4, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 24, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 20, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 52296, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 215420 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-16T20:44:23.749 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:23.750 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.751 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.752 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-16T20:44:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-16T20:44:23.755 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.756 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.759 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.760 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 1, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 1, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 0, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 1, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 1601, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-16T20:44:23.761 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:23.762 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:23.763 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 197503, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 55, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:23.764 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 164, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 14608, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 44, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-16T20:44:23.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 771, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 771, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 771, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 322.617416510, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.418440228 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-16T20:44:23.766 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 771, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 772 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 771, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 771, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 467, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 302, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-16T20:44:23.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-16T20:44:23.769 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 92, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 2, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 6, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 6, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-16T20:44:23.770 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-16T20:44:23.771 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 990, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1122975, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 990, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:23.772 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 990, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1122975, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:23.773 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 771, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 51270, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 771, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 51270, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:23.774 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 771, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 771, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 771, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 771, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-16T20:44:23.776 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x55f7b74f2960": { 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:23.777 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:23.778 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:44:24.582 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-16T20:44:24.582 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-16T20:44:24.582 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-16T20:44:24.582 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-16T20:44:24.582 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-16T20:44:24.582 DEBUG:tasks.check_counter:Getting stats from j 2024-03-16T20:44:24.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-16T20:44:24.830 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T20:44:25.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:25 smithi189 ceph-mon[36655]: pgmap v609: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:25.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:25 smithi023 ceph-mon[31879]: pgmap v609: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:25.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:25 smithi031 ceph-mon[36281]: pgmap v609: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:26.547 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 189, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 182, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 739104, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 108048, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.031345246, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.010877924, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.017694493, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.005188120, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 182, 2024-03-16T20:44:26.548 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.010305988, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000056626 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 739104, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 108048 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 137, 2024-03-16T20:44:26.549 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 160, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 101504, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 3, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.010114314, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007698762, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001146898, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 135, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008572892, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000063502 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:26.550 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 101504 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-16T20:44:26.551 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.000357372, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000293737, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.552 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:26.553 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:26.554 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:26.555 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148967, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 29, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:26.556 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 186, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 709914, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:26.557 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 186, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 186, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 709914, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-16T20:44:26.558 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-16T20:44:27.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:27 smithi023 ceph-mon[31879]: pgmap v610: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:27.332 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-16T20:44:27.332 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-16T20:44:27.332 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-16T20:44:27.332 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-16T20:44:27.332 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-16T20:44:27.332 DEBUG:tasks.check_counter:Getting stats from b 2024-03-16T20:44:27.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-16T20:44:27.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:27 smithi189 ceph-mon[36655]: pgmap v610: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:27.676 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:44:27.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:27 smithi031 ceph-mon[36281]: pgmap v610: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:29.565 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:29 smithi031 ceph-mon[36281]: pgmap v611: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:29.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:29 smithi023 ceph-mon[31879]: pgmap v611: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:29.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:29 smithi189 ceph-mon[36655]: pgmap v611: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2942, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 1699, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1146565, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 576301, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 9, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.154235590, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.055407644, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.130403777, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.004390604, 2024-03-16T20:44:29.948 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1697, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.162937864, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000096015 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 4384, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000839098, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000191 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 304, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 267, 2024-03-16T20:44:29.949 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1009833, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 183010, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 13, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 8, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.040380812, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.012613587, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.029516429, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.004676313, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 266, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.038688097, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000145443 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 58, 2024-03-16T20:44:29.950 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000016190, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000279 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 985824, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 110416 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 33, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 162, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 22685, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 333503, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.010792091, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.005521104, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003623740, 2024-03-16T20:44:29.951 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000820854, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 161, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.017546176, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000108982 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 10, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000001668, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000166 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 310352 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-16T20:44:29.952 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 83, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.008380395, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000100968 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 17, 2024-03-16T20:44:29.953 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000526592, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000030976 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 15, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 4, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 4, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.006723613, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001680903 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 11, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:29.954 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 11, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 20, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 2, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 18, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 13, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 169, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 3, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 218, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 102, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 11, 2024-03-16T20:44:29.955 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 99, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 2, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 2, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 4, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 324, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 13424234229, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-03-16T20:44:29.956 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 2, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 4, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 2, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:29.957 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 2, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 4 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 6, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 286, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 190, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 177, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 6, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 2, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 2, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:29.958 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 2, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-03-16T20:44:29.959 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 26, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 26, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 2, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-03-16T20:44:29.960 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4240187, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 19, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.363445753, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.019128723 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-03-16T20:44:29.961 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 24, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 111, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 87, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 23, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 119, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 96, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 20, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 195, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 175, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 6, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 6, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 53836, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 223612 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-03-16T20:44:29.962 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 21, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 7, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 15, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 10, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 18, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.963 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.964 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.965 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.966 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.967 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000862348, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000431174 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-03-16T20:44:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.969 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.970 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.971 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.005861265, 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002930632 2024-03-16T20:44:29.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 2, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 1, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 1, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-03-16T20:44:29.973 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 3, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 3, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 423, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:29.974 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:29.975 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 317957, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 122, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:29.976 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 164, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 96173, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 376, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-16T20:44:29.977 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 75, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 58448, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 27, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 75, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 75, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1.330975048, 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.017746333 2024-03-16T20:44:29.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 75, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 101 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 75, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 1, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 74, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 0, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 10, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 1, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 20, 2024-03-16T20:44:29.979 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 11, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 16, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 20, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-03-16T20:44:29.980 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-03-16T20:44:29.981 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 92, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 10, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 10, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-03-16T20:44:29.982 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 20, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 3 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-03-16T20:44:29.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 4, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 2, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 4, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 2, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 3275, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1868875, 2024-03-16T20:44:29.984 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 3275, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 3275, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1868875, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:29.985 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 75, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 58666, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 75, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 58666, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 75, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 75, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 75, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 75, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.987 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 2, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 218, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 2, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 218, 2024-03-16T20:44:29.988 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x55e26e4fc6e0": { 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 26, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 45883, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 26, 2024-03-16T20:44:29.989 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 19, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 45883, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:29.990 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-16T20:44:30.736 INFO:tasks.check_counter:Daemon mds.b mds.exported=2 2024-03-16T20:44:30.736 INFO:tasks.check_counter:Daemon mds.b mds.imported=2 2024-03-16T20:44:30.736 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=6 2024-03-16T20:44:30.736 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=286 2024-03-16T20:44:30.736 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-03-16T20:44:30.736 DEBUG:tasks.check_counter:Getting stats from e 2024-03-16T20:44:30.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-16T20:44:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:44:31.494 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:31 smithi031 ceph-mon[36281]: pgmap v612: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:31.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:31 smithi023 ceph-mon[31879]: pgmap v612: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:31.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:31 smithi189 ceph-mon[36655]: pgmap v612: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:33.498 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:33 smithi031 ceph-mon[36281]: pgmap v613: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:33.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:33 smithi023 ceph-mon[31879]: pgmap v613: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:33.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:33 smithi189 ceph-mon[36655]: pgmap v613: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 614, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 614, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 236966, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 190222, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.058148717, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.017069600, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.015333690, 2024-03-16T20:44:33.827 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.019049200, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 613, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.017824416, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000029077 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 709, 2024-03-16T20:44:33.828 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 668, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1094502, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 258693, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.071276036, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.018205656, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.032045947, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.021869216, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 667, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.027612449, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000041397 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.829 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 976400, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 109408 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 595, 2024-03-16T20:44:33.830 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 731, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 199744, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 495342, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.064740735, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.021477444, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.015301036, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.022521782, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 729, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.036051011, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000049452 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:33.831 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 310608 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2862, 2024-03-16T20:44:33.832 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.095070420, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000033218 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 0, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 0, 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-03-16T20:44:33.833 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 5, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 5, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 5, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 1, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 1, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-03-16T20:44:33.835 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:33.836 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 0 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 0, 2024-03-16T20:44:33.837 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:33.838 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 39, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 0, 2024-03-16T20:44:33.839 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4267947, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 4266296, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 39 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.840 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 4, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 24, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 20, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 2, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 19, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 17, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 5, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 31, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 26, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 0, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 0, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 48172, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 215420 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.841 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.842 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-03-16T20:44:33.843 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-03-16T20:44:33.846 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-03-16T20:44:33.847 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.848 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.849 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.851 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:33.852 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 2, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 1, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 2, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 0, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 0, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 3534, 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:33.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 189417, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 52, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 164, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 15312, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 48, 2024-03-16T20:44:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 1684, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 21, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 3, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 1684, 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-03-16T20:44:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1684, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 31.765435032, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.018863084 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1684, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1685 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 1684, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 1684, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 0, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 1072, 2024-03-16T20:44:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 610, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 1, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-03-16T20:44:33.861 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 92, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 5, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 5, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-03-16T20:44:33.862 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 2, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 0 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:33.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 1914, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1344196, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 1914, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1914, 2024-03-16T20:44:33.864 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1344196, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:33.865 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 1684, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 73648, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1684, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 73648, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:33.866 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 1684, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 1684, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1684, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1684, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:33.867 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:33.868 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x5569f298c960": { 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:33.870 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-16T20:44:34.837 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-16T20:44:34.837 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-16T20:44:34.837 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-16T20:44:34.837 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-16T20:44:34.837 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-16T20:44:34.837 DEBUG:tasks.check_counter:Getting stats from h 2024-03-16T20:44:34.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-16T20:44:35.235 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:44:35.481 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:35 smithi031 ceph-mon[36281]: pgmap v614: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:35.521 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:35 smithi189 ceph-mon[36655]: pgmap v614: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:35.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:35 smithi023 ceph-mon[31879]: pgmap v614: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 100, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 113, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 42177, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 166409, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 15, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.015177874, 2024-03-16T20:44:37.017 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.003612258, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.007877212, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.001843843, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 111, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.036961084, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000332982 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 64, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000019030, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000297 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.018 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 301, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 257, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1014802, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 153919, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 7, 2024-03-16T20:44:37.019 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.033850800, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.008421114, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.024608475, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.005234888, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 257, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.024082424, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000093705 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 31, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000005470, 2024-03-16T20:44:37.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000176 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 986608, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 114144 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2921, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 1825, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1147150, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 854328, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:37.022 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.155064965, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.058653887, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.132064305, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.003307135, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1823, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.118020907, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000064739 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 4404, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000852701, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000193 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.023 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 312032 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 97, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.010502407, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000108272 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.024 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 19, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000889610, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000046821 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 25, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 9, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 9, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.026247035, 2024-03-16T20:44:37.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002916337 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 16, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 11, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 20, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 3, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 17, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 13, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 169, 2024-03-16T20:44:37.026 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 1, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 1, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 3, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 230, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 107, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 16, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 98, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 3, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 3, 2024-03-16T20:44:37.027 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 8, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 324, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 13424234229, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 3, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 7, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 3, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 2, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 8 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 9, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 284, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 189, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 175, 2024-03-16T20:44:37.029 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 9, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 4, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 4, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:37.030 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 3, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 39, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 39, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 2, 2024-03-16T20:44:37.031 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4267947, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 24, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.431909605, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.017996233 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2024-03-16T20:44:37.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 24, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 113, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 89, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 23, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 118, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 95, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 20, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 201, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 181, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 1, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 12, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 11, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 54144, 2024-03-16T20:44:37.033 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 223612 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 35, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 7, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 25, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 11, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 21, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-03-16T20:44:37.034 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-03-16T20:44:37.035 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000629523, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000629523 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000097879, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000097879 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.036 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-03-16T20:44:37.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.038 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.004892526, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001630842 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.040 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.041 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-03-16T20:44:37.042 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-03-16T20:44:37.043 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-03-16T20:44:37.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 4, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.020627107, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.005156776 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 2, 2024-03-16T20:44:37.045 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 1, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 1, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 7, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 7, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 508, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:37.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:37.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:37.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 318258, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 123, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 164, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-16T20:44:37.049 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 96573, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 382, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 85, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 86960, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 28, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 85, 2024-03-16T20:44:37.050 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 85, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1.304975222, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.015352649 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 85, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 112 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 85, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 1, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 84, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 0, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 10, 2024-03-16T20:44:37.051 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 1, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 25, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 11, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 20, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 20, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-03-16T20:44:37.052 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-03-16T20:44:37.053 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 92, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 11, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 11, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 22, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 3 2024-03-16T20:44:37.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 8, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 4, 2024-03-16T20:44:37.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 8, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 4, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 3318, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1890031, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 3318, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 3318, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1890031, 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:37.056 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 85, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 87396, 2024-03-16T20:44:37.057 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 85, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 87396, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:37.058 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 85, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 85, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 85, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 85, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 4, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 436, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 4, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 436, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x559dead446e0": { 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 39, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 73643, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 39, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 24, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 73643, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:37.061 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:37.062 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:37.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:37.062 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-16T20:44:37.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:37 smithi023 ceph-mon[31879]: pgmap v615: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:37.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:37 smithi031 ceph-mon[36281]: pgmap v615: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:37.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:37 smithi189 ceph-mon[36655]: pgmap v615: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:37.869 INFO:tasks.check_counter:Daemon mds.h mds.exported=3 2024-03-16T20:44:37.869 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-03-16T20:44:37.869 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=9 2024-03-16T20:44:37.869 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=284 2024-03-16T20:44:37.869 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-16T20:44:37.869 DEBUG:tasks.check_counter:Getting stats from k 2024-03-16T20:44:37.870 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-16T20:44:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T20:44:39.527 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:39 smithi031 ceph-mon[36281]: pgmap v616: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:39.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:39 smithi023 ceph-mon[31879]: pgmap v616: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:39.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:39 smithi189 ceph-mon[36655]: pgmap v616: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2930, 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 1738, 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1143143, 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 572207, 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 9, 2024-03-16T20:44:40.004 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.150207595, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.054121564, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.127721506, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.004163897, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1736, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.146789275, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000084556 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 4373, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000863202, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000197 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:40.005 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 300, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 272, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 989770, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 191037, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 12, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 7, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.041520241, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.014170793, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.029371268, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.004739847, 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:40.006 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 271, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.034105175, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000125849 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 39, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000057119, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000001464 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 968704, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 116880 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 44, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 181, 2024-03-16T20:44:40.007 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 25685, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 341693, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.011359337, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.005851782, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003750174, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000874321, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 180, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.034500277, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000191668 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.008 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 18, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000003185, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000176 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 313184 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 75, 2024-03-16T20:44:40.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.009960698, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000132809 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 14, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000701086, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000050077 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 7, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 7, 2024-03-16T20:44:40.010 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 7, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.017412608, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002487515 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 11, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 20, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 4, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 16, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:40.011 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 13, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 170, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 1, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 1, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 3, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 209, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 104, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 99, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 2, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-03-16T20:44:40.012 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 2, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 6, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 324, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 13424234229, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:40.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 4, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 5, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 2, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 3, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 6 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 6, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 284, 2024-03-16T20:44:40.014 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 189, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 176, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 6, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 3, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 3, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 2, 2024-03-16T20:44:40.015 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 27, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 27, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 2, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-03-16T20:44:40.016 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4245569, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 17, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.255138302, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.015008135 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 24, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 112, 2024-03-16T20:44:40.017 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 88, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 23, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 118, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 95, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 20, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 199, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 179, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 1, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 9, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 8, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 51208, 2024-03-16T20:44:40.018 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 223612 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 13, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 7, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 7, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 7, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 18, 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-03-16T20:44:40.019 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001263451, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001263451 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-03-16T20:44:40.021 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000105922, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000105922 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:40.022 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-03-16T20:44:40.025 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001378223, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000689111 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.026 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:40.027 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.028 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.029 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-03-16T20:44:40.030 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.014665012, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.004888337 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 1, 2024-03-16T20:44:40.031 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 1, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 5, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 5, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 510, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:40.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:40.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:40.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 300611, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 111, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:40.035 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 164, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 96541, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 378, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-03-16T20:44:40.036 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 76, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 64061, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 33, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 76, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 76, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 291.772810874, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 3.839115932 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 76, 2024-03-16T20:44:40.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 103 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 76, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 1, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 75, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 0, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 10, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 1, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 18, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 11, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-03-16T20:44:40.038 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 18, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 20, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-03-16T20:44:40.039 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-03-16T20:44:40.040 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 92, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 10, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 10, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 22, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 3 2024-03-16T20:44:40.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 6, 2024-03-16T20:44:40.042 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 3, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 6, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 3, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 3270, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1848393, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 3270, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:40.043 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 3270, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1848393, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:40.044 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 76, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 64388, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 76, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 64388, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:40.045 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 76, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 76, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 76, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 76, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.046 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 3, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 327, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 3, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 327, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x55f287a4a6e0": { 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 27, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 51265, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 27, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 17, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 51265, 2024-03-16T20:44:40.048 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-03-16T20:44:40.049 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-03-16T20:44:40.758 INFO:tasks.check_counter:Daemon mds.k mds.exported=2 2024-03-16T20:44:40.758 INFO:tasks.check_counter:Daemon mds.k mds.imported=2 2024-03-16T20:44:40.758 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=6 2024-03-16T20:44:40.758 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=284 2024-03-16T20:44:40.758 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-16T20:44:40.758 DEBUG:tasks.check_counter:Getting stats from c 2024-03-16T20:44:40.759 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-16T20:44:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:44:41.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:41 smithi023 ceph-mon[31879]: pgmap v617: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:41.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:41 smithi189 ceph-mon[36655]: pgmap v617: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:41.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:41 smithi031 ceph-mon[36281]: pgmap v617: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:43.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:43 smithi023 ceph-mon[31879]: pgmap v618: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:43.597 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:43 smithi189 ceph-mon[36655]: pgmap v618: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:43.597 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-16T20:44:43.597 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:43.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 990, 2024-03-16T20:44:43.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 992, 2024-03-16T20:44:43.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 21279205, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 307900, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 7, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.091636241, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.021536381, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.034246325, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.029026700, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 990, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.061089009, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000061706 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 598, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 557, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 28556965, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 224395, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 5, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.077720219, 2024-03-16T20:44:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.015468538, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.049490000, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.015154447, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 556, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.033533543, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000060312 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 976832, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 110192 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 636, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 771, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 21137777, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 510674, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.072648415, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.019280949, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.028486536, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.020590178, 2024-03-16T20:44:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 770, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.039859511, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000051765 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 314432 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3336, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.074589826, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000022359 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 956, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 956, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 956, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-03-16T20:44:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 4, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 234, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-03-16T20:44:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-03-16T20:44:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-03-16T20:44:43.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 32115, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 37, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 4, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 73621139, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 73617381, 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-03-16T20:44:43.611 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 32115 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 24, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 890, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 866, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 19, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 120, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 101, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 956, 2024-03-16T20:44:43.612 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 1527, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 571, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 52176, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215420 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.613 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-03-16T20:44:43.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-03-16T20:44:43.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.617 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.623 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 4, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 3, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 2, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-03-16T20:44:43.624 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 3467, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:43.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:43.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 223471, 2024-03-16T20:44:43.627 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 76, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 164, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:43.628 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 751297, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 4739, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-03-16T20:44:43.629 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 1988, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 26, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 4, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 1988, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1988, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 47.995038344, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.024142373 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1988, 2024-03-16T20:44:43.630 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1989 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 1988, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 1988, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 1173, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 812, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-03-16T20:44:43.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-03-16T20:44:43.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-03-16T20:44:43.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 92, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 2, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 9, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 9, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-03-16T20:44:43.634 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-03-16T20:44:43.635 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 2220, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 70759293, 2024-03-16T20:44:43.636 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 2220, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2220, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 70759293, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:43.637 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1988, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 69439493, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1988, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 69439493, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.638 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1988, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1988, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1988, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1988, 2024-03-16T20:44:43.639 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:43.640 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x55f43d6ce960": { 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:43.641 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:43.642 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-16T20:44:43.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:43 smithi031 ceph-mon[36281]: pgmap v618: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:44.716 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-16T20:44:44.717 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-16T20:44:44.717 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-16T20:44:44.717 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-16T20:44:44.717 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-16T20:44:44.717 DEBUG:tasks.check_counter:Getting stats from f 2024-03-16T20:44:44.717 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-16T20:44:44.982 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:44:45.426 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:45 smithi189 ceph-mon[36655]: pgmap v619: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:45.534 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:45 smithi031 ceph-mon[36281]: pgmap v619: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:45.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:45 smithi023 ceph-mon[31879]: pgmap v619: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:47.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:47 smithi189 ceph-mon[36655]: pgmap v620: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:47.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:47 smithi023 ceph-mon[31879]: pgmap v620: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 1828, 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1828, 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 564744, 2024-03-16T20:44:47.634 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 567719, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.146168903, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.035605926, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.038710818, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.063015643, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1827, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.074845668, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000040966 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.635 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 233, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 192, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 977500, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 111340, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 3, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-03-16T20:44:47.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.028742885, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.008885706, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.020601654, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.004144170, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 191, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.014924994, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000078141 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 976944, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 110656 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 140, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 428, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 314471, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 4, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.007035750, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.004818688, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.001315220, 2024-03-16T20:44:47.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000024174, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 138, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.004943044, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000035819 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 314160 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.639 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3043, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.074130481, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000024360 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:47.640 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.641 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 3, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 3, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 3, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-03-16T20:44:47.642 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 1, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-03-16T20:44:47.643 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:47.644 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:47.645 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-03-16T20:44:47.646 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 26, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-03-16T20:44:47.647 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4240187, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 4238536, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.648 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 26 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 4, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 19, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 15, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 2, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 16, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 14, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 3, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 21, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 18, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-03-16T20:44:47.649 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 47412, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215420 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.650 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.651 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.652 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.655 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.655 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-03-16T20:44:47.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 1993, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-03-16T20:44:47.663 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:47.664 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:47.665 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 181320, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 47, 2024-03-16T20:44:47.666 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 164, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-03-16T20:44:47.667 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 13904, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 40, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 1827, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 21, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 3, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 1827, 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-03-16T20:44:47.668 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1827, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 17.896643698, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.009795645 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1827, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1828 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 1827, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 1827, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 1183, 2024-03-16T20:44:47.669 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 642, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 1, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-03-16T20:44:47.670 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-03-16T20:44:47.671 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 92, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 2, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 4, 2024-03-16T20:44:47.672 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 4, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.673 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:47.674 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 2059, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1342502, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 2059, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2059, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1342502, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1827, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 45888, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1827, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 45888, 2024-03-16T20:44:47.676 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1827, 2024-03-16T20:44:47.677 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1827, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1827, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1827, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:47.678 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x563639666960": { 2024-03-16T20:44:47.679 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:47.680 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:47.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:47.681 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.681 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:47.681 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-16T20:44:47.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:47 smithi031 ceph-mon[36281]: pgmap v620: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:48.530 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-16T20:44:48.530 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-16T20:44:48.530 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-16T20:44:48.530 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-16T20:44:48.530 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-16T20:44:48.530 DEBUG:tasks.check_counter:Getting stats from i 2024-03-16T20:44:48.530 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-16T20:44:48.847 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:44:49.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:49 smithi023 ceph-mon[31879]: pgmap v621: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:49.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:49 smithi031 ceph-mon[36281]: pgmap v621: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:49.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:49 smithi189 ceph-mon[36655]: pgmap v621: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 6468, 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 8905, 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 2707949, 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 24264888, 2024-03-16T20:44:50.678 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 9, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.628442972, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.242795009, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.221089438, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.143171986, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 8910, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.285066626, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000144227 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 8345, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001629149, 2024-03-16T20:44:50.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000195 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 999824, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 117664 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 3928, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 5289, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1183948, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 22379135, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 7, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.426260411, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.188359256, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.122992493, 2024-03-16T20:44:50.680 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.098839424, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 5288, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.830337196, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000346130 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4201, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000854498, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000203 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 324528 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 37058, 2024-03-16T20:44:50.681 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 42234, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 11385638, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 41746877, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 20, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 8, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 2.501323476, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.931323450, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 2.188728288, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.179114361, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 42226, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 13.279843501, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000314494 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.682 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 33069, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.011077750, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000334 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 84368, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 31872 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3809, 2024-03-16T20:44:50.683 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 16.237685601, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.004262978 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4554, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.320803754, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000070444 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 2624, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 2617, 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-03-16T20:44:50.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2617, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 20.727197219, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.007920212 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 7, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 21, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 1, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 98, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 5, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 10, 2024-03-16T20:44:50.685 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 83, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 90, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 25, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 1, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 1, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 3649, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 3382, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 45, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-03-16T20:44:50.686 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 7, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 18, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 7, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 4, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 234, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:50.687 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 14, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 338, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 3903, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 6, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 29566, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 28119, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 104, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 1339 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 361, 2024-03-16T20:44:50.688 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 688, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 6, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 924, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 857, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 61, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:50.689 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 7, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 1, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 32115, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 32115, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 37, 2024-03-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 37, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 4, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 73621139, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2120, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 97.928972551, 2024-03-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.046192911 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 29, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 895, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 866, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 24, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 125, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 101, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 98, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 1600, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 1502, 2024-03-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 1, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 886, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 885, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 71172, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 231804 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 3613, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 16, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 2624, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 28, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 88, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 764, 2024-03-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.809878917, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001060050 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 97, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.036795852, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000379338 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-03-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 61, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.058641668, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000961338 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 330, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.175598861, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000532117 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.698 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-03-16T20:44:50.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 105, 2024-03-16T20:44:50.698 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.166960349, 2024-03-16T20:44:50.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001590098 2024-03-16T20:44:50.698 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 48, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.014879599, 2024-03-16T20:44:50.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000309991 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 174, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.222257629, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.007024469 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 113, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 13.502372372, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.119490020 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:50.700 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 99, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 3.209453570, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.032418722 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.701 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 8, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.065335968, 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.008166996 2024-03-16T20:44:50.702 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.012225087, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.003056271 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.703 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.022604820, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.002511646 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 13, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.008693249, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000668711 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.704 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 792, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1.421499278, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001794822 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-03-16T20:44:50.705 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 4, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 3, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 642, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 642, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 521, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:50.706 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:50.707 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:50.708 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 291951, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 135, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 1056, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 12, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:50.709 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 164, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 176401, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 2266, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-03-16T20:44:50.710 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 7660, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 69802954, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 37, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 7660, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 7660, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 419.191587626, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.054724750 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 7660, 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 7927 2024-03-16T20:44:50.711 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 7660, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 27, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 7633, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 11, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 27, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 2147, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 48, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 1, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 5216, 2024-03-16T20:44:50.712 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 22, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 6, 2024-03-16T20:44:50.713 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-03-16T20:44:50.714 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 92, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 7, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 12, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 12, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 323, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 126 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.715 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 27, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 9, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 4688, 2024-03-16T20:44:50.716 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 857, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 47450, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 11504323, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 47450, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 47450, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 11504323, 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:50.717 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:50.718 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 7660, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 69895527, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 7660, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 69895527, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:50.719 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 7660, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 7660, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 7660, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 7660, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:50.720 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 857, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 92573, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 857, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 27, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 92573, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.721 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x559e117646e0": { 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 32115, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 69426835, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 32115, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2120, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 69426835, 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:50.722 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:50.723 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:50.723 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:50.723 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.723 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:50.723 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-16T20:44:51.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:51 smithi189 ceph-mon[36655]: pgmap v622: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:51.494 INFO:tasks.check_counter:Daemon mds.i mds.exported=7 2024-03-16T20:44:51.495 INFO:tasks.check_counter:Daemon mds.i mds.imported=7 2024-03-16T20:44:51.495 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=361 2024-03-16T20:44:51.495 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-03-16T20:44:51.495 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=1 2024-03-16T20:44:51.495 DEBUG:tasks.check_counter:Getting stats from l 2024-03-16T20:44:51.495 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b shell --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-16T20:44:51.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:51 smithi023 ceph-mon[31879]: pgmap v622: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:51.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:51 smithi031 ceph-mon[36281]: pgmap v622: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:51.753 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/mon.c/config 2024-03-16T20:44:53.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:53 smithi023 ceph-mon[31879]: pgmap v623: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:53.648 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:53 smithi189 ceph-mon[36655]: pgmap v623: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:53.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:53 smithi031 ceph-mon[36281]: pgmap v623: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-16T20:44:53.905 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-16T20:44:53.905 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-03-16T20:44:53.905 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 65, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 69, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 28493, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 80931, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 15, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.009439500, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.002036563, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.005667622, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000676867, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 67, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.014645047, 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000218582 2024-03-16T20:44:53.906 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 59, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000014370, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000243 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2512, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 3472 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 24, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 166, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 14798, 2024-03-16T20:44:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 338554, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 8, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.008851896, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.005303910, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.002382481, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000337807, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 165, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.016319723, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000098907 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 9, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000001291, 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000143 2024-03-16T20:44:53.908 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 318688 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 3137, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1853, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 2095297, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 634859, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.185425293, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.075318170, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.152409691, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.006719626, 2024-03-16T20:44:53.909 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1852, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.087321333, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000047149 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4321, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000714121, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000165 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 979232, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 119200 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-03-16T20:44:53.910 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 44, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001548761, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000035199 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000008274, 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000004137 2024-03-16T20:44:53.911 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 21, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 3, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002385091, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000795030 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 18, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-03-16T20:44:53.912 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 20, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 3, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 17, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 13, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 170, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 3, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 219, 2024-03-16T20:44:53.913 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 106, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 16, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 97, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-03-16T20:44:53.914 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 324, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 13424234229, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-03-16T20:44:53.915 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 286, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 190, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 181, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-03-16T20:44:53.916 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-03-16T20:44:53.917 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 5, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 5, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-03-16T20:44:53.918 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4195732, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.006057941, 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001514485 2024-03-16T20:44:53.919 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 24, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 109, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 85, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 23, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 121, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 98, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 20, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 190, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 170, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 3, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 3, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 50232, 2024-03-16T20:44:53.920 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223612 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 24, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 7, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 21, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 10, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 15, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001782131, 2024-03-16T20:44:53.921 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001782131 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000602960, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000301480 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.922 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-03-16T20:44:53.923 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-03-16T20:44:53.925 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.926 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.927 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.928 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.929 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.930 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-03-16T20:44:53.931 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 2, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 2, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 447, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-03-16T20:44:53.932 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-03-16T20:44:53.933 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-03-16T20:44:53.934 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 301930, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 116, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-03-16T20:44:53.935 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 8600, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 164, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 96205, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 380, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 42, 2024-03-16T20:44:53.936 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 11955, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 42, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 42, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.211995701, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.005047516 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 42, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 66 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 42, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 0, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 42, 2024-03-16T20:44:53.937 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 4, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 7, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 6, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-03-16T20:44:53.938 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-03-16T20:44:53.939 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 92, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 2, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 9, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 9, 2024-03-16T20:44:53.940 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 17, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 2 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-03-16T20:44:53.941 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:53.942 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 3222, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1832042, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 3222, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 3222, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1832042, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:53.943 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 42, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 11955, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 42, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 11955, 2024-03-16T20:44:53.944 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 42, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 42, 2024-03-16T20:44:53.945 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 42, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 42, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-03-16T20:44:53.946 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x5579957de6e0": { 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-03-16T20:44:53.947 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 5, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1428, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 5, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1428, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.948 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-03-16T20:44:53.949 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-16T20:44:54.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:54 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:44:54.613 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:54 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:44:54.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:54 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T20:44:54.974 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-16T20:44:54.974 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-16T20:44:54.974 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-16T20:44:54.974 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=286 2024-03-16T20:44:54.974 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-16T20:44:54.974 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-16T20:44:54.987 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-16T20:44:54.987 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:44:54.987 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T20:44:55.015 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-16T20:44:55.015 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-16T20:44:55.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:44:55.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- 172.21.15.23:0/1107093395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 msgr2=0x7fc8ec109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 --2- 172.21.15.23:0/1107093395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec109950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009ca0 tx=0x7fc8e402c0f0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- 172.21.15.23:0/1107093395 shutdown_connections 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 --2- 172.21.15.23:0/1107093395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec109950 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 --2- 172.21.15.23:0/1107093395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec102700 0x7fc8ec102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 --2- 172.21.15.23:0/1107093395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec101510 0x7fc8ec101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- 172.21.15.23:0/1107093395 >> 172.21.15.23:0/1107093395 conn(0x7fc8ec0fcaa0 msgr2=0x7fc8ec0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- 172.21.15.23:0/1107093395 shutdown_connections 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- 172.21.15.23:0/1107093395 wait complete. 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 Processor -- start 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.215+0000 7fc8f4457640 1 -- start start 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 0x7fc8ec197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec102700 0x7fc8ec197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec19f020 con 0x7fc8ec1030c0 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec19f190 con 0x7fc8ec101510 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec19f350 con 0x7fc8ec102700 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53064/0 (socket says 172.21.15.23:53064) 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f21f3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 0x7fc8ec197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 -- 172.21.15.23:0/2399404198 learned_addr learned my addr 172.21.15.23:0/2399404198 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f19f2640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec102700 0x7fc8ec197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 -- 172.21.15.23:0/2399404198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec102700 msgr2=0x7fc8ec197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec102700 0x7fc8ec197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 -- 172.21.15.23:0/2399404198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 msgr2=0x7fc8ec197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 0x7fc8ec197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 -- 172.21.15.23:0/2399404198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009900 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f21f3640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 0x7fc8ec197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f29f4640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4037040 tx=0x7fc8e4009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40329b0 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8e4003e50 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec19f5d0 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4040450 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.216+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec19fad0 con 0x7fc8ec1030c0 2024-03-16T20:44:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.217+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec108830 con 0x7fc8ec1030c0 2024-03-16T20:44:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.218+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc8e400a040 con 0x7fc8ec1030c0 2024-03-16T20:44:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.219+0000 7fc8e37fe640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 0x7fc8d80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.219+0000 7fc8f21f3640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 0x7fc8d80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.219+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fc8e40f0450 con 0x7fc8ec1030c0 2024-03-16T20:44:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.219+0000 7fc8f21f3640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 0x7fc8d80a3f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec102560 tx=0x7fc8e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.224+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc8e40b93b0 con 0x7fc8ec1030c0 2024-03-16T20:44:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.407+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ec02d000 con 0x7fc8ec1030c0 2024-03-16T20:44:55.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.409+0000 7fc8e37fe640 1 -- 172.21.15.23:0/2399404198 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7fc8e4034e60 con 0x7fc8ec1030c0 2024-03-16T20:44:55.412 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:44:55.413 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:44:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 msgr2=0x7fc8d80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 0x7fc8d80a3f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec102560 tx=0x7fc8e8005f90 comp rx=0 tx=0).stop 2024-03-16T20:44:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 msgr2=0x7fc8ec19eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4037040 tx=0x7fc8e4009360 comp rx=0 tx=0).stop 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 shutdown_connections 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fc8d80a1ab0 0x7fc8d80a3f70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8ec1030c0 0x7fc8ec19eae0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8ec102700 0x7fc8ec197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 --2- 172.21.15.23:0/2399404198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8ec101510 0x7fc8ec197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 >> 172.21.15.23:0/2399404198 conn(0x7fc8ec0fcaa0 msgr2=0x7fc8ec105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 shutdown_connections 2024-03-16T20:44:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.411+0000 7fc8f4457640 1 -- 172.21.15.23:0/2399404198 wait complete. 2024-03-16T20:44:55.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:55 smithi023 ceph-mon[31879]: pgmap v624: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:55.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:44:55.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:44:55.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:55 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:44:55.426 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T20:29:10.202841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-16T20:33:47.324782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-16T20:33:48.405221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-16T20:33:52.101031+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-16T20:44:55.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 -- 172.21.15.23:0/3777597728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb8101500 msgr2=0x7fbdb8101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 --2- 172.21.15.23:0/3777597728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb8101500 0x7fbdb8101920 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009900 tx=0x7fbda802bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 -- 172.21.15.23:0/3777597728 shutdown_connections 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 --2- 172.21.15.23:0/3777597728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 --2- 172.21.15.23:0/3777597728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb81026f0 0x7fbdb8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 --2- 172.21.15.23:0/3777597728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb8101500 0x7fbdb8101920 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 -- 172.21.15.23:0/3777597728 >> 172.21.15.23:0/3777597728 conn(0x7fbdb80fca90 msgr2=0x7fbdb80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 -- 172.21.15.23:0/3777597728 shutdown_connections 2024-03-16T20:44:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.529+0000 7fbdbd015640 1 -- 172.21.15.23:0/3777597728 wait complete. 2024-03-16T20:44:55.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 Processor -- start 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 -- start start 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb8101500 0x7fbdb80752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8074580 con 0x7fbdb81026f0 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb80746f0 con 0x7fbdb81030b0 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdbd015640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81a1680 con 0x7fbdb8101500 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdb659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdb659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53078/0 (socket says 172.21.15.23:53078) 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdb659c640 1 -- 172.21.15.23:0/2312854815 learned_addr learned my addr 172.21.15.23:0/2312854815 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdb759e640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.530+0000 7fbdb6d9d640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb8101500 0x7fbdb80752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 -- 172.21.15.23:0/2312854815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb8101500 msgr2=0x7fbdb80752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb8101500 0x7fbdb80752a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 -- 172.21.15.23:0/2312854815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 msgr2=0x7fbdb8073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 -- 172.21.15.23:0/2312854815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0009840 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb759e640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdb659c640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00ea20 tx=0x7fbdac00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac009720 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbdac005500 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda8009560 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac010430 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.531+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81a1c60 con 0x7fbdb81026f0 2024-03-16T20:44:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.532+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb8108890 con 0x7fbdb81026f0 2024-03-16T20:44:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.533+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbdac005000 con 0x7fbdb81026f0 2024-03-16T20:44:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.533+0000 7fbd9ffff640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 0x7fbda00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.534+0000 7fbdb6d9d640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 0x7fbda00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.534+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fbdac014070 con 0x7fbdb81026f0 2024-03-16T20:44:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.534+0000 7fbdb6d9d640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 0x7fbda00a3ef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005ab0 tx=0x7fbda8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.538+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbdac096950 con 0x7fbdb81026f0 2024-03-16T20:44:55.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:55 smithi031 ceph-mon[36281]: pgmap v624: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:55.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:44:55.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:44:55.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:55 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:44:55.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:55 smithi189 ceph-mon[36655]: pgmap v624: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-16T20:44:55.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T20:44:55.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T20:44:55.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:55 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' 2024-03-16T20:44:55.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.720+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdb802cf90 con 0x7fbdb81026f0 2024-03-16T20:44:55.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.720+0000 7fbd9ffff640 1 -- 172.21.15.23:0/2312854815 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdac09a010 con 0x7fbdb81026f0 2024-03-16T20:44:55.724 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 msgr2=0x7fbda00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 0x7fbda00a3ef0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005ab0 tx=0x7fbda8005fd0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 msgr2=0x7fbdb80738e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00ea20 tx=0x7fbdac00ee20 comp rx=0 tx=0).stop 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 shutdown_connections 2024-03-16T20:44:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbda00a1a30 0x7fbda00a3ef0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdb81030b0 0x7fbdb8073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb81026f0 0x7fbdb80738e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 --2- 172.21.15.23:0/2312854815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdb8101500 0x7fbdb80752a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.722+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 >> 172.21.15.23:0/2312854815 conn(0x7fbdb80fca90 msgr2=0x7fbdb8105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.723+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 shutdown_connections 2024-03-16T20:44:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.723+0000 7fbdbd015640 1 -- 172.21.15.23:0/2312854815 wait complete. 2024-03-16T20:44:55.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- 172.21.15.23:0/3051740100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 msgr2=0x7ff108074330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 --2- 172.21.15.23:0/3051740100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff108074330 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009900 tx=0x7ff0fc02bcd0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- 172.21.15.23:0/3051740100 shutdown_connections 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 --2- 172.21.15.23:0/3051740100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 0x7ff10810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 --2- 172.21.15.23:0/3051740100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff108074330 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 --2- 172.21.15.23:0/3051740100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 0x7ff108073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- 172.21.15.23:0/3051740100 >> 172.21.15.23:0/3051740100 conn(0x7ff1080fca90 msgr2=0x7ff1080feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- 172.21.15.23:0/3051740100 shutdown_connections 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- 172.21.15.23:0/3051740100 wait complete. 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 Processor -- start 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.840+0000 7ff10e684640 1 -- start start 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 0x7ff108199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 0x7ff1081a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081a1290 con 0x7ff108073ed0 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081a1400 con 0x7ff1081093b0 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081a15c0 con 0x7ff1080754b0 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53096/0 (socket says 172.21.15.23:53096) 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 -- 172.21.15.23:0/4208959942 learned_addr learned my addr 172.21.15.23:0/4208959942 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff1077fe640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 0x7ff108199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10cc21640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 0x7ff1081a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 -- 172.21.15.23:0/4208959942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 msgr2=0x7ff108199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 0x7ff108199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 -- 172.21.15.23:0/4208959942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 msgr2=0x7ff1081a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 0x7ff1081a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 -- 172.21.15.23:0/4208959942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc009560 con 0x7ff108073ed0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff107fff640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800e9f0 tx=0x7ff0f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8009700 con 0x7ff108073ed0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0f800cd80 con 0x7ff108073ed0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8018920 con 0x7ff108073ed0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1081a18a0 con 0x7ff108073ed0 2024-03-16T20:44:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.841+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1081a1d80 con 0x7ff108073ed0 2024-03-16T20:44:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.842+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff108108170 con 0x7ff108073ed0 2024-03-16T20:44:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.843+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff0f8018a80 con 0x7ff108073ed0 2024-03-16T20:44:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.844+0000 7ff1057fa640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 0x7ff0f00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.844+0000 7ff1077fe640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 0x7ff0f00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.844+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7ff0f8014070 con 0x7ff108073ed0 2024-03-16T20:44:55.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.844+0000 7ff1077fe640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 0x7ff0f00a3ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009230 tx=0x7ff0fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:55.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:55.847+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff0f8096ed0 con 0x7ff108073ed0 2024-03-16T20:44:56.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.026+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff10810f2a0 con 0x7ff108073ed0 2024-03-16T20:44:56.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.027+0000 7ff1057fa640 1 -- 172.21.15.23:0/4208959942 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff0f809a590 con 0x7ff108073ed0 2024-03-16T20:44:56.030 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:44:56.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 msgr2=0x7ff0f00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 0x7ff0f00a3ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009230 tx=0x7ff0fc005fd0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 msgr2=0x7ff1081997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800e9f0 tx=0x7ff0f800eeb0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 shutdown_connections 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7ff0f00a1a30 0x7ff0f00a3ef0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1081093b0 0x7ff1081a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1080754b0 0x7ff108199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 --2- 172.21.15.23:0/4208959942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff108073ed0 0x7ff1081997a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 >> 172.21.15.23:0/4208959942 conn(0x7ff1080fca90 msgr2=0x7ff108105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 shutdown_connections 2024-03-16T20:44:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.029+0000 7ff10e684640 1 -- 172.21.15.23:0/4208959942 wait complete. 2024-03-16T20:44:56.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 -- 172.21.15.23:0/1193775702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c1015f0 msgr2=0x7f2d5c101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 --2- 172.21.15.23:0/1193775702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c101a70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009900 tx=0x7f2d5002bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 -- 172.21.15.23:0/1193775702 shutdown_connections 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 --2- 172.21.15.23:0/1193775702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 0x7f2d5c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 --2- 172.21.15.23:0/1193775702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c101a70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 --2- 172.21.15.23:0/1193775702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c100400 0x7f2d5c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 -- 172.21.15.23:0/1193775702 >> 172.21.15.23:0/1193775702 conn(0x7f2d5c0fb990 msgr2=0x7f2d5c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 -- 172.21.15.23:0/1193775702 shutdown_connections 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.146+0000 7f2d61b72640 1 -- 172.21.15.23:0/1193775702 wait complete. 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 Processor -- start 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 -- start start 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 0x7f2d5c19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c19f0a0 con 0x7f2d5c100400 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c19f210 con 0x7f2d5c1015f0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d61b72640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c19f3d0 con 0x7f2d5c101fb0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53116/0 (socket says 172.21.15.23:53116) 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.147+0000 7f2d5b7fe640 1 -- 172.21.15.23:0/3064676773 learned_addr learned my addr 172.21.15.23:0/3064676773 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5affd640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5bfff640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 0x7f2d5c19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 -- 172.21.15.23:0/3064676773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 msgr2=0x7f2d5c19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 0x7f2d5c19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 -- 172.21.15.23:0/3064676773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c1015f0 msgr2=0x7f2d5c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 -- 172.21.15.23:0/3064676773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c009490 con 0x7f2d5c100400 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d5b7fe640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e9c0 tx=0x7f2d4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c00cde0 con 0x7f2d5c100400 2024-03-16T20:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d50009560 con 0x7f2d5c100400 2024-03-16T20:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d4c010910 con 0x7f2d5c100400 2024-03-16T20:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c018960 con 0x7f2d5c100400 2024-03-16T20:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.148+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c19fa50 con 0x7f2d5c100400 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.149+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c107720 con 0x7f2d5c100400 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.150+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2d4c010a80 con 0x7f2d5c100400 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.151+0000 7f2d58ff9640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 0x7f2d440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.151+0000 7f2d5affd640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 0x7f2d440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.151+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f2d4c014070 con 0x7f2d5c100400 2024-03-16T20:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.151+0000 7f2d5affd640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 0x7f2d440a3ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009230 tx=0x7f2d50005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.155+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2d4c096bc0 con 0x7f2d5c100400 2024-03-16T20:44:56.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.340+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2d5c02cf90 con 0x7f2d5c100400 2024-03-16T20:44:56.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.340+0000 7f2d58ff9640 1 -- 172.21.15.23:0/3064676773 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2d4c09a280 con 0x7f2d5c100400 2024-03-16T20:44:56.344 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 msgr2=0x7f2d440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 0x7f2d440a3ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009230 tx=0x7f2d50005fd0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 msgr2=0x7f2d5c06a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e9c0 tx=0x7f2d4c00ee80 comp rx=0 tx=0).stop 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 shutdown_connections 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f2d440a1a30 0x7f2d440a3ef0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d5c101fb0 0x7f2d5c19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d5c1015f0 0x7f2d5c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 --2- 172.21.15.23:0/3064676773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d5c100400 0x7f2d5c06a650 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 >> 172.21.15.23:0/3064676773 conn(0x7f2d5c0fb990 msgr2=0x7f2d5c1048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 shutdown_connections 2024-03-16T20:44:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.342+0000 7f2d61b72640 1 -- 172.21.15.23:0/3064676773 wait complete. 2024-03-16T20:44:56.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 -- 172.21.15.23:0/2532096136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 msgr2=0x7f3a70109630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 --2- 172.21.15.23:0/2532096136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a70109630 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009ca0 tx=0x7f3a6802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 -- 172.21.15.23:0/2532096136 shutdown_connections 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 --2- 172.21.15.23:0/2532096136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a70109630 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 --2- 172.21.15.23:0/2532096136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70106d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 --2- 172.21.15.23:0/2532096136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 0x7f3a701043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 -- 172.21.15.23:0/2532096136 >> 172.21.15.23:0/2532096136 conn(0x7f3a700fb990 msgr2=0x7f3a700fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 -- 172.21.15.23:0/2532096136 shutdown_connections 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.459+0000 7f3a7613a640 1 -- 172.21.15.23:0/2532096136 wait complete. 2024-03-16T20:44:56.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 Processor -- start 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 -- start start 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 0x7f3a70197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7019f070 con 0x7f3a70107240 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7019f1e0 con 0x7f3a70104910 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a7613a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7019f3a0 con 0x7f3a70101fe0 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53136/0 (socket says 172.21.15.23:53136) 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 -- 172.21.15.23:0/819998201 learned_addr learned my addr 172.21.15.23:0/819998201 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 0x7f3a70197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.23:52092/0 (socket says 172.21.15.23:52092) 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 -- 172.21.15.23:0/819998201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 msgr2=0x7f3a70197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 0x7f3a70197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 -- 172.21.15.23:0/819998201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 msgr2=0x7f3a70197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6ffff640 1 -- 172.21.15.23:0/819998201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a64009490 con 0x7f3a70107240 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.460+0000 7f3a6effd640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a6ffff640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3a68037040 tx=0x7f3a68009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680329b0 con 0x7f3a70107240 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a68003e50 con 0x7f3a70107240 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68040450 con 0x7f3a70107240 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a68009900 con 0x7f3a70107240 2024-03-16T20:44:56.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.461+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7019f940 con 0x7f3a70107240 2024-03-16T20:44:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.462+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a700fd480 con 0x7f3a70107240 2024-03-16T20:44:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.463+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3a6800a040 con 0x7f3a70107240 2024-03-16T20:44:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.463+0000 7f3a6cff9640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 0x7f3a580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.463+0000 7f3a6f7fe640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 0x7f3a580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.463+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f3a680f0900 con 0x7f3a70107240 2024-03-16T20:44:56.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.464+0000 7f3a6f7fe640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 0x7f3a580a3ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009bd0 tx=0x7f3a60005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.469 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2399404198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2312854815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:56 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4208959942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.467+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3a680b9860 con 0x7f3a70107240 2024-03-16T20:44:56.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2399404198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:56.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2312854815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:56 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4208959942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.649+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3a7010cf10 con 0x7f3a70107240 2024-03-16T20:44:56.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.650+0000 7f3a6cff9640 1 -- 172.21.15.23:0/819998201 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3a68034e60 con 0x7f3a70107240 2024-03-16T20:44:56.653 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-16T20:44:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.651+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 msgr2=0x7f3a580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.651+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 0x7f3a580a3ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009bd0 tx=0x7f3a60005f90 comp rx=0 tx=0).stop 2024-03-16T20:44:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.651+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 msgr2=0x7f3a7019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.651+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3a68037040 tx=0x7f3a68009360 comp rx=0 tx=0).stop 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 shutdown_connections 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f3a580a1a30 0x7f3a580a3ef0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a70107240 0x7f3a7019eb30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a70104910 0x7f3a70197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 --2- 172.21.15.23:0/819998201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a70101fe0 0x7f3a70197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 >> 172.21.15.23:0/819998201 conn(0x7f3a700fb990 msgr2=0x7f3a701028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 shutdown_connections 2024-03-16T20:44:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.652+0000 7f3a7613a640 1 -- 172.21.15.23:0/819998201 wait complete. 2024-03-16T20:44:56.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-16T20:44:56.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2399404198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:56.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2312854815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:56 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4208959942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-16T20:44:56.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- 172.21.15.23:0/346898875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 msgr2=0x7f92e81021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- 172.21.15.23:0/346898875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e81021b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009900 tx=0x7f92e402bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- 172.21.15.23:0/346898875 shutdown_connections 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- 172.21.15.23:0/346898875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 0x7f92e8105520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- 172.21.15.23:0/346898875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 0x7f92e8104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- 172.21.15.23:0/346898875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e81021b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- 172.21.15.23:0/346898875 >> 172.21.15.23:0/346898875 conn(0x7f92e80fb9a0 msgr2=0x7f92e80fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- 172.21.15.23:0/346898875 shutdown_connections 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- 172.21.15.23:0/346898875 wait complete. 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 Processor -- start 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- start start 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 0x7f92e819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 0x7f92e81a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81a56c0 con 0x7f92e80ffd80 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81a5830 con 0x7f92e81050a0 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.768+0000 7f92ef86b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81a59f0 con 0x7f92e81026f0 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53162/0 (socket says 172.21.15.23:53162) 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 -- 172.21.15.23:0/3559274484 learned_addr learned my addr 172.21.15.23:0/3559274484 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ede08640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 0x7f92e81a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ece06640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 0x7f92e819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 -- 172.21.15.23:0/3559274484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 msgr2=0x7f92e819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 0x7f92e819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 -- 172.21.15.23:0/3559274484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 msgr2=0x7f92e81a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 0x7f92e81a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 -- 172.21.15.23:0/3559274484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4009560 con 0x7f92e80ffd80 2024-03-16T20:44:56.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ed607640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006010 tx=0x7f92e4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40043d0 con 0x7f92e80ffd80 2024-03-16T20:44:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92e40052d0 con 0x7f92e80ffd80 2024-03-16T20:44:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e81a5c70 con 0x7f92e80ffd80 2024-03-16T20:44:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4003cc0 con 0x7f92e80ffd80 2024-03-16T20:44:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.769+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e81a6180 con 0x7f92e80ffd80 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.770+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e80fd490 con 0x7f92e80ffd80 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.772+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f92e4005440 con 0x7f92e80ffd80 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.773+0000 7f92de7fc640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 0x7f92d40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.773+0000 7f92ece06640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 0x7f92d40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.773+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f92e40eff70 con 0x7f92e80ffd80 2024-03-16T20:44:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.773+0000 7f92ece06640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 0x7f92d40a3ef0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92e819f0e0 tx=0x7f92d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:56.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.776+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f92e40b8e40 con 0x7f92e80ffd80 2024-03-16T20:44:56.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.972+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f92e81136a0 con 0x7f92e80ffd80 2024-03-16T20:44:56.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.973+0000 7f92de7fc640 1 -- 172.21.15.23:0/3559274484 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) v1 ==== 81+0+592 (secure 0 0 0) 0x7f92e40bc500 con 0x7f92e80ffd80 2024-03-16T20:44:56.976 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:44:56.976 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1269138528 2024-03-17T20:26:23.333931+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4262074056 2024-03-17T20:26:49.856928+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2140563160 2024-03-17T20:26:49.856928+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4012523374 2024-03-17T20:26:49.856928+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1525116937 2024-03-17T20:26:23.333931+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3890527981 2024-03-17T20:26:23.333931+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3223781845 2024-03-17T20:26:23.333931+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2899821830 2024-03-17T20:26:49.856928+0000 2024-03-16T20:44:56.977 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-16T20:44:56.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 msgr2=0x7f92d40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 0x7f92d40a3ef0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92e819f0e0 tx=0x7f92d8005f90 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 msgr2=0x7f92e819dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006010 tx=0x7f92e4005e80 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 shutdown_connections 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f92d40a1a30 0x7f92d40a3ef0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92e81050a0 0x7f92e81a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e81026f0 0x7f92e819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 --2- 172.21.15.23:0/3559274484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92e80ffd80 0x7f92e819dbd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 >> 172.21.15.23:0/3559274484 conn(0x7f92e80fb9a0 msgr2=0x7f92e81034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 shutdown_connections 2024-03-16T20:44:56.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:56.975+0000 7f92ef86b640 1 -- 172.21.15.23:0/3559274484 wait complete. 2024-03-16T20:44:56.989 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-16T20:44:56.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:44:56.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-16T20:44:57.085 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T20:44:57.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:44:57.085 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-16T20:44:57.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:44:57.120 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-16T20:44:57.224 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-16T20:44:57.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:44:57.224 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-16T20:44:57.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-16T20:44:57.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-16T20:44:57.350 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T20:44:57.360 INFO:tasks.cephadm:Teardown begin 2024-03-16T20:44:57.360 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-16T20:44:57.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 -- 172.21.15.23:0/2682776522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 msgr2=0x7fbebc101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/2682776522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc101a80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009900 tx=0x7fbeac02bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 -- 172.21.15.23:0/2682776522 shutdown_connections 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/2682776522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc101fc0 0x7fbebc108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/2682776522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc101a80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/2682776522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc100410 0x7fbebc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 -- 172.21.15.23:0/2682776522 >> 172.21.15.23:0/2682776522 conn(0x7fbebc0fb980 msgr2=0x7fbebc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 -- 172.21.15.23:0/2682776522 shutdown_connections 2024-03-16T20:44:57.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.463+0000 7fbec1f2c640 1 -- 172.21.15.23:0/2682776522 wait complete. 2024-03-16T20:44:57.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 Processor -- start 2024-03-16T20:44:57.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 -- start start 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 0x7fbebc1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc101fc0 0x7fbebc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc19f090 con 0x7fbebc101600 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc19f200 con 0x7fbebc100410 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.464+0000 7fbec1f2c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc19f3c0 con 0x7fbebc101fc0 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 0x7fbebc1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53176/0 (socket says 172.21.15.23:53176) 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 -- 172.21.15.23:0/3201462108 learned_addr learned my addr 172.21.15.23:0/3201462108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebbfff640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc101fc0 0x7fbebc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 -- 172.21.15.23:0/3201462108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc101fc0 msgr2=0x7fbebc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc101fc0 0x7fbebc19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 -- 172.21.15.23:0/3201462108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 msgr2=0x7fbebc1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 0x7fbebc1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 -- 172.21.15.23:0/3201462108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac009560 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebb7fe640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 0x7fbebc1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbebaffd640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006010 tx=0x7fbeac005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0043d0 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbeac0052d0 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac003cc0 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc19f640 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.465+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc19fba0 con 0x7fbebc101600 2024-03-16T20:44:57.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.466+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc107800 con 0x7fbebc101600 2024-03-16T20:44:57.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.467+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbeac005440 con 0x7fbebc101600 2024-03-16T20:44:57.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.468+0000 7fbeb8ff9640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 0x7fbea40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.468+0000 7fbebb7fe640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 0x7fbea40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.468+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7fbeac0ef8d0 con 0x7fbebc101600 2024-03-16T20:44:57.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.468+0000 7fbebb7fe640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 0x7fbea40a3f70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbebc101460 tx=0x7fbeb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.472+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbeac0b87a0 con 0x7fbebc101600 2024-03-16T20:44:57.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:57 smithi023 ceph-mon[31879]: pgmap v625: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:57.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3064676773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/819998201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:57 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3559274484' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:44:57.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:57 smithi031 ceph-mon[36281]: pgmap v625: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:57.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3064676773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/819998201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:57 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3559274484' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:44:57.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.697+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbebc10cf20 con 0x7fbebc101600 2024-03-16T20:44:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.697+0000 7fbeb8ff9640 1 -- 172.21.15.23:0/3201462108 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+11454 (secure 0 0 0) 0x7fbeac0bbe60 con 0x7fbebc101600 2024-03-16T20:44:57.701 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:44:57.701 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:43:54.672912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":149,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24493,24416,24418,24511,24497]},"id":1}]} 2024-03-16T20:44:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2024-03-16T20:44:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 msgr2=0x7fbea40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 0x7fbea40a3f70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbebc101460 tx=0x7fbeb0005f90 comp rx=0 tx=0).stop 2024-03-16T20:44:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 msgr2=0x7fbebc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006010 tx=0x7fbeac005e80 comp rx=0 tx=0).stop 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 shutdown_connections 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fbea40a1ab0 0x7fbea40a3f70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbebc101fc0 0x7fbebc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101600 0x7fbebc197ae0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 --2- 172.21.15.23:0/3201462108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbebc100410 0x7fbebc1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 >> 172.21.15.23:0/3201462108 conn(0x7fbebc0fb980 msgr2=0x7fbebc1049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 shutdown_connections 2024-03-16T20:44:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.700+0000 7fbec1f2c640 1 -- 172.21.15.23:0/3201462108 wait complete. 2024-03-16T20:44:57.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-16T20:44:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:57 smithi189 ceph-mon[36655]: pgmap v625: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-16T20:44:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3064676773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/819998201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-16T20:44:57.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:57 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3559274484' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T20:44:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 -- 172.21.15.23:0/3246304699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 msgr2=0x7f1b741008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/3246304699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741008b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009860 tx=0x7f1b6402bb70 comp rx=0 tx=0).stop 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 -- 172.21.15.23:0/3246304699 shutdown_connections 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/3246304699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 0x7f1b741088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/3246304699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/3246304699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741008b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 -- 172.21.15.23:0/3246304699 >> 172.21.15.23:0/3246304699 conn(0x7f1b740fba20 msgr2=0x7f1b740fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 -- 172.21.15.23:0/3246304699 shutdown_connections 2024-03-16T20:44:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.816+0000 7f1b78ed4640 1 -- 172.21.15.23:0/3246304699 wait complete. 2024-03-16T20:44:57.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 Processor -- start 2024-03-16T20:44:57.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 -- start start 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 0x7f1b7419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419f0a0 con 0x7f1b74101680 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419f210 con 0x7f1b74100490 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b78ed4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7419f3d0 con 0x7f1b74102040 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b71d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b71d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53190/0 (socket says 172.21.15.23:53190) 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b71d9b640 1 -- 172.21.15.23:0/2118069099 learned_addr learned my addr 172.21.15.23:0/2118069099 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b72d9d640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 0x7f1b7419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 -- 172.21.15.23:0/2118069099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 msgr2=0x7f1b7419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 0x7f1b7419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 -- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 msgr2=0x7f1b74197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b7259c640 1 -- 172.21.15.23:0/2118069099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68009560 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.817+0000 7f1b71d9b640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b7259c640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741975b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b6402c0b0 tx=0x7f1b640045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6402c510 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b6402cb50 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b64009490 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64032600 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.818+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7419f830 con 0x7f1b74100490 2024-03-16T20:44:57.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.819+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b741077b0 con 0x7f1b74100490 2024-03-16T20:44:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.820+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1b6403b040 con 0x7f1b74100490 2024-03-16T20:44:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.820+0000 7f1b637fe640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 0x7f1b5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.820+0000 7f1b71d9b640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 0x7f1b5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.820+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f1b640ef220 con 0x7f1b74100490 2024-03-16T20:44:57.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.821+0000 7f1b71d9b640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 0x7f1b5c0a3ef0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b74198ac0 tx=0x7f1b68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:57.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:57.824+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1b640b81a0 con 0x7f1b74100490 2024-03-16T20:44:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.047+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7410cf30 con 0x7f1b74100490 2024-03-16T20:44:58.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.049+0000 7f1b637fe640 1 -- 172.21.15.23:0/2118069099 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+11454 (secure 0 0 0) 0x7f1b6404b090 con 0x7f1b74100490 2024-03-16T20:44:58.052 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:44:58.052 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/2314600527","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":2314600527},{"type":"v1","addr":"172.21.15.23:6835","nonce":2314600527}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1010201356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1010201356},{"type":"v1","addr":"172.21.15.23:6837","nonce":1010201356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":59,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-16T20:33:49.239658+0000","modified":"2024-03-16T20:43:54.672912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24511,"mds_1":24497,"mds_2":24493,"mds_3":24418,"mds_4":24416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":149,"addr":"172.21.15.23:6839/2750734224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2750734224},{"type":"v1","addr":"172.21.15.23:6839","nonce":2750734224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/4113023748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":4113023748},{"type":"v1","addr":"172.21.15.23:6841","nonce":4113023748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24416":{"gid":24416,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":15,"addr":"172.21.15.31:6833/1919282692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1919282692},{"type":"v1","addr":"172.21.15.31:6833","nonce":1919282692}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24418":{"gid":24418,"name":"l","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6833/1083314774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1083314774},{"type":"v1","addr":"172.21.15.189:6833","nonce":1083314774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6835/354681253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":354681253},{"type":"v1","addr":"172.21.15.31:6835","nonce":354681253}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1658699703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1658699703},{"type":"v1","addr":"172.21.15.189:6835","nonce":1658699703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3204120502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3204120502},{"type":"v1","addr":"172.21.15.189:6837","nonce":3204120502}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24493":{"gid":24493,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6837/424329834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":424329834},{"type":"v1","addr":"172.21.15.31:6837","nonce":424329834}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24497":{"gid":24497,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.31:6839/37663771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":37663771},{"type":"v1","addr":"172.21.15.31:6839","nonce":37663771}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.189:6839/2333060283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2333060283},{"type":"v1","addr":"172.21.15.189:6839","nonce":2333060283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24511,"qdb_cluster":[24493,24416,24418,24511,24497]},"id":1}]} 2024-03-16T20:44:58.053 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 msgr2=0x7f1b5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 0x7f1b5c0a3ef0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b74198ac0 tx=0x7f1b68009260 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 msgr2=0x7f1b741975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741975b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b6402c0b0 tx=0x7f1b640045a0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 shutdown_connections 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f1b5c0a1a30 0x7f1b5c0a3ef0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b74102040 0x7f1b7419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b74101680 0x7f1b74197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 --2- 172.21.15.23:0/2118069099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b74100490 0x7f1b741975b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 >> 172.21.15.23:0/2118069099 conn(0x7f1b740fba20 msgr2=0x7f1b74104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 shutdown_connections 2024-03-16T20:44:58.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.051+0000 7f1b78ed4640 1 -- 172.21.15.23:0/2118069099 wait complete. 2024-03-16T20:44:58.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-16T20:44:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- 172.21.15.23:0/2619404655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 msgr2=0x7f450c1043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 --2- 172.21.15.23:0/2619404655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1043e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4508009ca0 tx=0x7f450802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- 172.21.15.23:0/2619404655 shutdown_connections 2024-03-16T20:44:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 --2- 172.21.15.23:0/2619404655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c107250 0x7f450c109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 --2- 172.21.15.23:0/2619404655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c104920 0x7f450c106d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 --2- 172.21.15.23:0/2619404655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1043e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- 172.21.15.23:0/2619404655 >> 172.21.15.23:0/2619404655 conn(0x7f450c0fb980 msgr2=0x7f450c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- 172.21.15.23:0/2619404655 shutdown_connections 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- 172.21.15.23:0/2619404655 wait complete. 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 Processor -- start 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.166+0000 7f4514647640 1 -- start start 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 0x7f450c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c107250 0x7f450c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c19f0d0 con 0x7f450c101ff0 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c19f240 con 0x7f450c104920 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c19f400 con 0x7f450c107250 2024-03-16T20:44:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45123e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45123e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53210/0 (socket says 172.21.15.23:53210) 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45123e3640 1 -- 172.21.15.23:0/2123402436 learned_addr learned my addr 172.21.15.23:0/2123402436 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 0x7f450c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4512be4640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c107250 0x7f450c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 -- 172.21.15.23:0/2123402436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c107250 msgr2=0x7f450c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c107250 0x7f450c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 -- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 msgr2=0x7f450c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 -- 172.21.15.23:0/2123402436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508009900 con 0x7f450c104920 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45123e3640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4511be2640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 0x7f450c197b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e9c0 tx=0x7f44fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00ccc0 con 0x7f450c104920 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44fc00ce20 con 0x7f450c104920 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c19f6e0 con 0x7f450c104920 2024-03-16T20:44:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc004a00 con 0x7f450c104920 2024-03-16T20:44:58.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.167+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c19fba0 con 0x7f450c104920 2024-03-16T20:44:58.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.168+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c0fd470 con 0x7f450c104920 2024-03-16T20:44:58.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.169+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f44fc005020 con 0x7f450c104920 2024-03-16T20:44:58.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.170+0000 7f45037fe640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 0x7f44f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.170+0000 7f45123e3640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 0x7f44f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.170+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f44fc014070 con 0x7f450c104920 2024-03-16T20:44:58.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.170+0000 7f45123e3640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 0x7f44f80a3ef0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4508004af0 tx=0x7f45080326d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.174+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f44fc096550 con 0x7f450c104920 2024-03-16T20:44:58.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.360+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f450c10cf20 con 0x7f450c104920 2024-03-16T20:44:58.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.361+0000 7f45037fe640 1 -- 172.21.15.23:0/2123402436 <== mon.2 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24561 (secure 0 0 0) 0x7f44fc099c10 con 0x7f450c104920 2024-03-16T20:44:58.365 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-16T20:44:58.365 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"3d249ea8-e3d3-11ee-95c9-87774f69a715","created":"2024-03-16T20:25:46.502894+0000","modified":"2024-03-16T20:34:51.409333+0000","last_up_change":"2024-03-16T20:32:58.109133+0000","last_in_change":"2024-03-16T20:32:37.757109+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T20:29:10.202841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-16T20:33:47.324782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-16T20:33:48.405221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-16T20:33:52.101031+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96865589-182e-46bf-b061-3b6f0521156b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6803","nonce":3821836785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6805","nonce":3821836785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6809","nonce":3821836785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3821836785},{"type":"v1","addr":"172.21.15.23:6807","nonce":3821836785}]},"public_addr":"172.21.15.23:6803/3821836785","cluster_addr":"172.21.15.23:6805/3821836785","heartbeat_back_addr":"172.21.15.23:6809/3821836785","heartbeat_front_addr":"172.21.15.23:6807/3821836785","state":["exists","up"]},{"osd":1,"uuid":"8c8d3f5e-f4ba-4019-938e-6c9a9c5037c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6801","nonce":639380907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6803","nonce":639380907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6807","nonce":639380907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":639380907},{"type":"v1","addr":"172.21.15.31:6805","nonce":639380907}]},"public_addr":"172.21.15.31:6801/639380907","cluster_addr":"172.21.15.31:6803/639380907","heartbeat_back_addr":"172.21.15.31:6807/639380907","heartbeat_front_addr":"172.21.15.31:6805/639380907","state":["exists","up"]},{"osd":2,"uuid":"07b0e4ef-221b-4cfa-b176-8ff89dda1c90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6801","nonce":3710574436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6803","nonce":3710574436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6807","nonce":3710574436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3710574436},{"type":"v1","addr":"172.21.15.189:6805","nonce":3710574436}]},"public_addr":"172.21.15.189:6801/3710574436","cluster_addr":"172.21.15.189:6803/3710574436","heartbeat_back_addr":"172.21.15.189:6807/3710574436","heartbeat_front_addr":"172.21.15.189:6805/3710574436","state":["exists","up"]},{"osd":3,"uuid":"cc0249a5-38ab-46a3-957f-f13367ec3060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6811","nonce":1075234291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6813","nonce":1075234291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6817","nonce":1075234291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1075234291},{"type":"v1","addr":"172.21.15.23:6815","nonce":1075234291}]},"public_addr":"172.21.15.23:6811/1075234291","cluster_addr":"172.21.15.23:6813/1075234291","heartbeat_back_addr":"172.21.15.23:6817/1075234291","heartbeat_front_addr":"172.21.15.23:6815/1075234291","state":["exists","up"]},{"osd":4,"uuid":"20ff34b3-508e-49ca-a5aa-61c2c52551b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6809","nonce":2047837401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6811","nonce":2047837401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6815","nonce":2047837401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2047837401},{"type":"v1","addr":"172.21.15.31:6813","nonce":2047837401}]},"public_addr":"172.21.15.31:6809/2047837401","cluster_addr":"172.21.15.31:6811/2047837401","heartbeat_back_addr":"172.21.15.31:6815/2047837401","heartbeat_front_addr":"172.21.15.31:6813/2047837401","state":["exists","up"]},{"osd":5,"uuid":"7b28071d-a593-49b1-9561-614bf17df712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6809","nonce":2617907376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6811","nonce":2617907376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6815","nonce":2617907376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2617907376},{"type":"v1","addr":"172.21.15.189:6813","nonce":2617907376}]},"public_addr":"172.21.15.189:6809/2617907376","cluster_addr":"172.21.15.189:6811/2617907376","heartbeat_back_addr":"172.21.15.189:6815/2617907376","heartbeat_front_addr":"172.21.15.189:6813/2617907376","state":["exists","up"]},{"osd":6,"uuid":"e7698cf3-f3a6-4e39-991a-b101bc81d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6819","nonce":103652673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6821","nonce":103652673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6825","nonce":103652673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":103652673},{"type":"v1","addr":"172.21.15.23:6823","nonce":103652673}]},"public_addr":"172.21.15.23:6819/103652673","cluster_addr":"172.21.15.23:6821/103652673","heartbeat_back_addr":"172.21.15.23:6825/103652673","heartbeat_front_addr":"172.21.15.23:6823/103652673","state":["exists","up"]},{"osd":7,"uuid":"82df9886-90ce-4a0a-8bdc-09b78caf46af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6817","nonce":2033067958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6819","nonce":2033067958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6823","nonce":2033067958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2033067958},{"type":"v1","addr":"172.21.15.31:6821","nonce":2033067958}]},"public_addr":"172.21.15.31:6817/2033067958","cluster_addr":"172.21.15.31:6819/2033067958","heartbeat_back_addr":"172.21.15.31:6823/2033067958","heartbeat_front_addr":"172.21.15.31:6821/2033067958","state":["exists","up"]},{"osd":8,"uuid":"49cd9654-b197-40a7-ab09-f7d720b898ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6817","nonce":3435219550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6819","nonce":3435219550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6823","nonce":3435219550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3435219550},{"type":"v1","addr":"172.21.15.189:6821","nonce":3435219550}]},"public_addr":"172.21.15.189:6817/3435219550","cluster_addr":"172.21.15.189:6819/3435219550","heartbeat_back_addr":"172.21.15.189:6823/3435219550","heartbeat_front_addr":"172.21.15.189:6821/3435219550","state":["exists","up"]},{"osd":9,"uuid":"38b7a821-6c63-4636-84e2-cfa25c440244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6827","nonce":1125641919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6829","nonce":1125641919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6833","nonce":1125641919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1125641919},{"type":"v1","addr":"172.21.15.23:6831","nonce":1125641919}]},"public_addr":"172.21.15.23:6827/1125641919","cluster_addr":"172.21.15.23:6829/1125641919","heartbeat_back_addr":"172.21.15.23:6833/1125641919","heartbeat_front_addr":"172.21.15.23:6831/1125641919","state":["exists","up"]},{"osd":10,"uuid":"6615be9c-9a48-448d-af63-51c8ab219057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6825","nonce":3304713388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6827","nonce":3304713388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6831","nonce":3304713388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3304713388},{"type":"v1","addr":"172.21.15.31:6829","nonce":3304713388}]},"public_addr":"172.21.15.31:6825/3304713388","cluster_addr":"172.21.15.31:6827/3304713388","heartbeat_back_addr":"172.21.15.31:6831/3304713388","heartbeat_front_addr":"172.21.15.31:6829/3304713388","state":["exists","up"]},{"osd":11,"uuid":"7877d354-a2b5-4c0e-844d-b56717ad5a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6825","nonce":702443898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6827","nonce":702443898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6831","nonce":702443898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":702443898},{"type":"v1","addr":"172.21.15.189:6829","nonce":702443898}]},"public_addr":"172.21.15.189:6825/702443898","cluster_addr":"172.21.15.189:6827/702443898","heartbeat_back_addr":"172.21.15.189:6831/702443898","heartbeat_front_addr":"172.21.15.189:6829/702443898","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:20.222899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:28:43.822156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:07.200120+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:33.226570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:29:57.338402+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:22.292600+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:30:47.749230+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:12.635099+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:31:37.574548+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:03.823486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:29.764382+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T20:32:55.877257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.b","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2899821830":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1269138528":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/4262074056":"2024-03-17T20:26:49.856928+0000","172.21.15.23:6801/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/2140563160":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/4012523374":"2024-03-17T20:26:49.856928+0000","172.21.15.23:0/1525116937":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/3890527981":"2024-03-17T20:26:23.333931+0000","172.21.15.23:0/3223781845":"2024-03-17T20:26:23.333931+0000","172.21.15.23:6800/2899821830":"2024-03-17T20:26:49.856928+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 msgr2=0x7f44f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 0x7f44f80a3ef0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4508004af0 tx=0x7f45080326d0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 msgr2=0x7f450c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 0x7f450c197b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e9c0 tx=0x7f44fc00ee80 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 shutdown_connections 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f44f80a1a30 0x7f44f80a3ef0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f450c107250 0x7f450c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f450c104920 0x7f450c197b20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 --2- 172.21.15.23:0/2123402436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f450c101ff0 0x7f450c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 >> 172.21.15.23:0/2123402436 conn(0x7f450c0fb980 msgr2=0x7f450c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 shutdown_connections 2024-03-16T20:44:58.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.363+0000 7f4514647640 1 -- 172.21.15.23:0/2123402436 wait complete. 2024-03-16T20:44:58.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 -- 172.21.15.23:0/1595442758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1015e0 msgr2=0x7f36bc101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 --2- 172.21.15.23:0/1595442758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1015e0 0x7f36bc101a60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009900 tx=0x7f36b002bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 -- 172.21.15.23:0/1595442758 shutdown_connections 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 --2- 172.21.15.23:0/1595442758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 0x7f36bc108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 --2- 172.21.15.23:0/1595442758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1015e0 0x7f36bc101a60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 --2- 172.21.15.23:0/1595442758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1003f0 0x7f36bc100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 -- 172.21.15.23:0/1595442758 >> 172.21.15.23:0/1595442758 conn(0x7f36bc0fb980 msgr2=0x7f36bc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 -- 172.21.15.23:0/1595442758 shutdown_connections 2024-03-16T20:44:58.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.478+0000 7f36c30b5640 1 -- 172.21.15.23:0/1595442758 wait complete. 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 Processor -- start 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 -- start start 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 0x7f36bc19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 0x7f36bc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19f0b0 con 0x7f36bc1003f0 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19f220 con 0x7f36bc101fa0 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c30b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc19f3e0 con 0x7f36bc1015e0 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53232/0 (socket says 172.21.15.23:53232) 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 -- 172.21.15.23:0/65735200 learned_addr learned my addr 172.21.15.23:0/65735200 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c1652640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 0x7f36bc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36bbfff640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 0x7f36bc19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 -- 172.21.15.23:0/65735200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 msgr2=0x7f36bc19c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 0x7f36bc19c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.479+0000 7f36c0e51640 1 -- 172.21.15.23:0/65735200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 msgr2=0x7f36bc19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36c0e51640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 0x7f36bc19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36c0e51640 1 -- 172.21.15.23:0/65735200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0009560 con 0x7f36bc1003f0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36bbfff640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 0x7f36bc19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36c0e51640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36b400eb70 tx=0x7f36b400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4003b70 con 0x7f36bc1003f0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36b400ee70 con 0x7f36bc1003f0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc19f6c0 con 0x7f36bc1003f0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b40107f0 con 0x7f36bc1003f0 2024-03-16T20:44:58.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.480+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc19fb80 con 0x7f36bc1003f0 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.481+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc107730 con 0x7f36bc1003f0 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.482+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f36b4004020 con 0x7f36bc1003f0 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.482+0000 7f36b9ffb640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 0x7f36a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.483+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9063+0+0 (secure 0 0 0) 0x7f36b4014070 con 0x7f36bc1003f0 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.483+0000 7f36bbfff640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 0x7f36a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.483+0000 7f36bbfff640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 0x7f36a40a3ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f36bc068f90 tx=0x7f36b0005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.487+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f36b4096cc0 con 0x7f36bc1003f0 2024-03-16T20:44:58.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3201462108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2118069099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3201462108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2118069099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.707+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f36bc10be00 con 0x7f36bc1003f0 2024-03-16T20:44:58.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.709+0000 7f36b9ffb640 1 -- 172.21.15.23:0/65735200 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) v1 ==== 161+0+0 (secure 0 0 0) 0x7f36b4003e60 con 0x7f36bc1003f0 2024-03-16T20:44:58.713 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 msgr2=0x7f36a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 0x7f36a40a3ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f36bc068f90 tx=0x7f36b0005be0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 msgr2=0x7f36bc069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36b400eb70 tx=0x7f36b400c6a0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 shutdown_connections 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f36a40a1a30 0x7f36a40a3ef0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36bc101fa0 0x7f36bc19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36bc1015e0 0x7f36bc19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 --2- 172.21.15.23:0/65735200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36bc1003f0 0x7f36bc069220 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 >> 172.21.15.23:0/65735200 conn(0x7f36bc0fb980 msgr2=0x7f36bc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 shutdown_connections 2024-03-16T20:44:58.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.712+0000 7f36c30b5640 1 -- 172.21.15.23:0/65735200 wait complete. 2024-03-16T20:44:58.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3201462108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.719 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2118069099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-16T20:44:58.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-16T20:44:58.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 -- 172.21.15.23:0/3338061119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 msgr2=0x7fe6381043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/3338061119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381043c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe628009900 tx=0x7fe62802bc30 comp rx=0 tx=0).stop 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 -- 172.21.15.23:0/3338061119 shutdown_connections 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/3338061119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638107230 0x7fe638109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/3338061119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638104900 0x7fe638106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/3338061119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381043c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 -- 172.21.15.23:0/3338061119 >> 172.21.15.23:0/3338061119 conn(0x7fe6380fb980 msgr2=0x7fe6380fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 -- 172.21.15.23:0/3338061119 shutdown_connections 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.837+0000 7fe63c8ed640 1 -- 172.21.15.23:0/3338061119 wait complete. 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 Processor -- start 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 -- start start 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 0x7fe638197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638107230 0x7fe63819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819f0c0 con 0x7fe638101fd0 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819f230 con 0x7fe638107230 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63c8ed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63819f3f0 con 0x7fe638104900 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53292/0 (socket says 172.21.15.23:53292) 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 -- 172.21.15.23:0/2184097412 learned_addr learned my addr 172.21.15.23:0/2184097412 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe636d9d640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638107230 0x7fe63819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe62ffff640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 0x7fe638197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 -- 172.21.15.23:0/2184097412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 msgr2=0x7fe638197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 0x7fe638197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 -- 172.21.15.23:0/2184097412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638107230 msgr2=0x7fe63819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638107230 0x7fe63819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 -- 172.21.15.23:0/2184097412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628009560 con 0x7fe638101fd0 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe62ffff640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 0x7fe638197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:44:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe63659c640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe628006010 tx=0x7fe6280040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.838+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280044c0 con 0x7fe638101fd0 2024-03-16T20:44:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.839+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6280052d0 con 0x7fe638101fd0 2024-03-16T20:44:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.839+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63819f670 con 0x7fe638101fd0 2024-03-16T20:44:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.839+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628003e10 con 0x7fe638101fd0 2024-03-16T20:44:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.839+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63819fb80 con 0x7fe638101fd0 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.840+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6380fd470 con 0x7fe638101fd0 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.841+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe628005440 con 0x7fe638101fd0 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.842+0000 7fe62f7fe640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 0x7fe6240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.842+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9493+0+0 (secure 0 0 0) 0x7fe6280f0270 con 0x7fe638101fd0 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.842+0000 7fe62ffff640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 0x7fe6240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.843+0000 7fe62ffff640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 0x7fe6240a3ef0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe620005d00 tx=0x7fe620005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:58.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:58.846+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe6280b8f90 con 0x7fe638101fd0 2024-03-16T20:44:59.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.066+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe63810cf00 con 0x7fe638101fd0 2024-03-16T20:44:59.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a[31875]: 2024-03-16T20:44:58.708+0000 7f9a58937700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-mds[70796]: -- 172.21.15.23:0/3964197118 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b2d6fe1a0 con 0x557b2d6d4800 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: starting mds.a at 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-mds[70440]: -- 172.21.15.23:0/1795118171 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c455921a0 con 0x562c45569000 2024-03-16T20:44:59.089 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:44:58 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: starting mds.g at 2024-03-16T20:44:59.187 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.187 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mds[66516]: -- 172.21.15.31:0/3841992749 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb73e121a0 con 0x55cb73de8c00 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: starting mds.h at 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mds[65880]: -- 172.21.15.31:0/2243155416 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d6e5ec01a0 con 0x55d6e5e96800 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: starting mds.k at 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.188 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mds[66838]: -- 172.21.15.31:0/1147114385 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e19b301a0 con 0x555e19b07000 2024-03-16T20:44:59.189 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: starting mds.b at 2024-03-16T20:44:59.189 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.189 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.189 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-mds[66205]: -- 172.21.15.31:0/2472873857 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557c9e921a0 con 0x5557c9e68800 2024-03-16T20:44:59.189 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:44:58 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: starting mds.e at 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mds[67175]: -- 172.21.15.189:0/3942366251 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556e46eea1a0 con 0x556e46ec1000 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: starting mds.f at 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mds[67483]: -- 172.21.15.189:0/4237889897 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560f782801a0 con 0x560f78256800 2024-03-16T20:44:59.220 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: starting mds.c at 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mds[67806]: -- 172.21.15.189:0/1599490879 <== mon.2 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56440ad781a0 con 0x56440ad4e800 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: starting mds.i at 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: ignoring --setuser ceph since I am not root 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: ignoring --setgroup ceph since I am not root 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mds[66862]: -- 172.21.15.189:0/4231202017 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a87ee35080 con 0x55a87edfe800 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-mds[66862]: -- 172.21.15.189:0/4231202017 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a87ee281a0 con 0x55a87edfe800 2024-03-16T20:44:59.221 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:44:58 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: starting mds.l at 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: pgmap v626: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2123402436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T20:44:59.589 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:44:59 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:44:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: pgmap v626: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:44:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2123402436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T20:44:59.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T20:44:59.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T20:44:59.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T20:44:59.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T20:44:59.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T20:44:59.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:44:59 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:44:59.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.715+0000 7fe62f7fe640 1 -- 172.21.15.23:0/2184097412 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6280bc650 con 0x7fe638101fd0 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: pgmap v626: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2123402436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/65735200' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: osdmap e93: 12 total, 12 up, 12 in 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-16T20:44:59.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:44:59 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 msgr2=0x7fe6240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 0x7fe6240a3ef0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe620005d00 tx=0x7fe620005ed0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 msgr2=0x7fe6381975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe628006010 tx=0x7fe6280040c0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 shutdown_connections 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe6240a1a30 0x7fe6240a3ef0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638107230 0x7fe63819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe638104900 0x7fe638197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 --2- 172.21.15.23:0/2184097412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe638101fd0 0x7fe6381975d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.718+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 >> 172.21.15.23:0/2184097412 conn(0x7fe6380fb980 msgr2=0x7fe638102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.719+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 shutdown_connections 2024-03-16T20:44:59.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.719+0000 7fe63c8ed640 1 -- 172.21.15.23:0/2184097412 wait complete. 2024-03-16T20:44:59.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-16T20:44:59.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.852+0000 7fe3863d7640 1 -- 172.21.15.23:0/2171326285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 msgr2=0x7fe380109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:59.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.852+0000 7fe3863d7640 1 --2- 172.21.15.23:0/2171326285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380109620 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe378009ca0 tx=0x7fe37802c0f0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- 172.21.15.23:0/2171326285 shutdown_connections 2024-03-16T20:44:59.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- 172.21.15.23:0/2171326285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380109620 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- 172.21.15.23:0/2171326285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3801048a0 0x7fe380106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- 172.21.15.23:0/2171326285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe380101f70 0x7fe380104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- 172.21.15.23:0/2171326285 >> 172.21.15.23:0/2171326285 conn(0x7fe3800fb980 msgr2=0x7fe3800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- 172.21.15.23:0/2171326285 shutdown_connections 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- 172.21.15.23:0/2171326285 wait complete. 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 Processor -- start 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- start start 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101f70 0x7fe3801018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3801048a0 0x7fe3800fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe380100bc0 con 0x7fe3801071d0 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe380100d30 con 0x7fe380101f70 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.853+0000 7fe3863d7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3801a15e0 con 0x7fe3801048a0 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:59.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53298/0 (socket says 172.21.15.23:53298) 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 -- 172.21.15.23:0/4082696219 learned_addr learned my addr 172.21.15.23:0/4082696219 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe37ffff640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101f70 0x7fe3801018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe37f7fe640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3801048a0 0x7fe3800fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 -- 172.21.15.23:0/4082696219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3801048a0 msgr2=0x7fe3800fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3801048a0 0x7fe3800fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 -- 172.21.15.23:0/4082696219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101f70 msgr2=0x7fe3801018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101f70 0x7fe3801018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 -- 172.21.15.23:0/4082696219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378009900 con 0x7fe3801071d0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe384974640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe3780098d0 tx=0x7fe378034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378032eb0 con 0x7fe3801071d0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3801a1830 con 0x7fe3801071d0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe378003e50 con 0x7fe3801071d0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3780406b0 con 0x7fe3801071d0 2024-03-16T20:44:59.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.854+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3801a1ce0 con 0x7fe3801071d0 2024-03-16T20:44:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.855+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3800fd470 con 0x7fe3801071d0 2024-03-16T20:44:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.857+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe37800a040 con 0x7fe3801071d0 2024-03-16T20:44:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.857+0000 7fe37d7fa640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 0x7fe3680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:44:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.857+0000 7fe37ffff640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 0x7fe3680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:44:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.858+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9493+0+0 (secure 0 0 0) 0x7fe3780f0e40 con 0x7fe3801071d0 2024-03-16T20:44:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.858+0000 7fe37ffff640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 0x7fe3680a3ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe370005fd0 tx=0x7fe370005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:44:59.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:44:59.861+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe3780b9d10 con 0x7fe3801071d0 2024-03-16T20:45:00.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.046+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe38010cf00 con 0x7fe3801071d0 2024-03-16T20:45:00.687 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: Cluster is now healthy 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6833/3532710658,v1:172.21.15.31:6837/3532710658] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6834/2140056246,v1:172.21.15.31:6838/2140056246] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6835/2289351093,v1:172.21.15.31:6839/2289351093] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6835/603811333,v1:172.21.15.189:6839/603811333] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6834/1520409704,v1:172.21.15.189:6838/1520409704] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.31:6832/1395033742,v1:172.21.15.31:6836/1395033742] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6832/34146954,v1:172.21.15.189:6837/34146954] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.189:6833/3477837885,v1:172.21.15.189:6836/3477837885] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6838/3146929225,v1:172.21.15.23:6840/3146929225] up:boot 2024-03-16T20:45:00.688 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: mds.? [v2:172.21.15.23:6839/2795540319,v1:172.21.15.23:6841/2795540319] up:boot 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: fsmap 12 up:standby 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:45:00.689 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:00 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: Cluster is now healthy 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6833/3532710658,v1:172.21.15.31:6837/3532710658] up:boot 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6834/2140056246,v1:172.21.15.31:6838/2140056246] up:boot 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6835/2289351093,v1:172.21.15.31:6839/2289351093] up:boot 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6835/603811333,v1:172.21.15.189:6839/603811333] up:boot 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6834/1520409704,v1:172.21.15.189:6838/1520409704] up:boot 2024-03-16T20:45:00.720 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.31:6832/1395033742,v1:172.21.15.31:6836/1395033742] up:boot 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6832/34146954,v1:172.21.15.189:6837/34146954] up:boot 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.189:6833/3477837885,v1:172.21.15.189:6836/3477837885] up:boot 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6838/3146929225,v1:172.21.15.23:6840/3146929225] up:boot 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: mds.? [v2:172.21.15.23:6839/2795540319,v1:172.21.15.23:6841/2795540319] up:boot 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: fsmap 12 up:standby 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:45:00.721 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:00 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:00.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.722+0000 7fe37d7fa640 1 -- 172.21.15.23:0/4082696219 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe3780bd3d0 con 0x7fe3801071d0 2024-03-16T20:45:00.726 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 msgr2=0x7fe3680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 0x7fe3680a3ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe370005fd0 tx=0x7fe370005850 comp rx=0 tx=0).stop 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 msgr2=0x7fe380100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe3780098d0 tx=0x7fe378034d40 comp rx=0 tx=0).stop 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 shutdown_connections 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fe3680a1a30 0x7fe3680a3ef0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3801071d0 0x7fe380100460 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3801048a0 0x7fe3800fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 --2- 172.21.15.23:0/4082696219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101f70 0x7fe3801018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 >> 172.21.15.23:0/4082696219 conn(0x7fe3800fb980 msgr2=0x7fe3801028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:45:00.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 shutdown_connections 2024-03-16T20:45:00.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.725+0000 7fe3863d7640 1 -- 172.21.15.23:0/4082696219 wait complete. 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: Cluster is now healthy 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/2184097412' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6833/3532710658,v1:172.21.15.31:6837/3532710658] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6834/2140056246,v1:172.21.15.31:6838/2140056246] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6835/2289351093,v1:172.21.15.31:6839/2289351093] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6835/603811333,v1:172.21.15.189:6839/603811333] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6834/1520409704,v1:172.21.15.189:6838/1520409704] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.31:6832/1395033742,v1:172.21.15.31:6836/1395033742] up:boot 2024-03-16T20:45:00.742 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6832/34146954,v1:172.21.15.189:6837/34146954] up:boot 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.189:6833/3477837885,v1:172.21.15.189:6836/3477837885] up:boot 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6838/3146929225,v1:172.21.15.23:6840/3146929225] up:boot 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: mds.? [v2:172.21.15.23:6839/2795540319,v1:172.21.15.23:6841/2795540319] up:boot 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: fsmap 12 up:standby 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='mgr.14152 172.21.15.23:0/627797300' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-16T20:45:00.743 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:00 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:00.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 -- 172.21.15.23:0/2585255077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 msgr2=0x7fd778106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/2585255077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778106d10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009900 tx=0x7fd76c02bc30 comp rx=0 tx=0).stop 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 -- 172.21.15.23:0/2585255077 shutdown_connections 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/2585255077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778107250 0x7fd778109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/2585255077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778106d10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/2585255077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778101ff0 0x7fd7781043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 -- 172.21.15.23:0/2585255077 >> 172.21.15.23:0/2585255077 conn(0x7fd7780fb980 msgr2=0x7fd7780fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 -- 172.21.15.23:0/2585255077 shutdown_connections 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.849+0000 7fd77e4ff640 1 -- 172.21.15.23:0/2585255077 wait complete. 2024-03-16T20:45:00.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 Processor -- start 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 -- start start 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 0x7fd778197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 0x7fd77819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77819f140 con 0x7fd778104920 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77819f2b0 con 0x7fd778101ff0 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77e4ff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77819f470 con 0x7fd778107250 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46464/0 (socket says 172.21.15.23:46464) 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 -- 172.21.15.23:0/169330393 learned_addr learned my addr 172.21.15.23:0/169330393 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd777fff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 0x7fd778197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77ca9c640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 0x7fd77819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:00.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 -- 172.21.15.23:0/169330393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 msgr2=0x7fd77819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 0x7fd77819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 -- 172.21.15.23:0/169330393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 msgr2=0x7fd778197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 0x7fd778197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd7777fe640 1 -- 172.21.15.23:0/169330393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c009560 con 0x7fd778104920 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd777fff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 0x7fd778197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.850+0000 7fd77ca9c640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 0x7fd77819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd7777fe640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c004df0 con 0x7fd778104920 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd76c005430 con 0x7fd778104920 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77819f6f0 con 0x7fd778104920 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c03e450 con 0x7fd778104920 2024-03-16T20:45:00.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.851+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77819fc00 con 0x7fd778104920 2024-03-16T20:45:00.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.852+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7780fd470 con 0x7fd778104920 2024-03-16T20:45:00.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.853+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd76c003680 con 0x7fd778104920 2024-03-16T20:45:00.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.853+0000 7fd7757fa640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 0x7fd7600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:00.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.854+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8997+0+0 (secure 0 0 0) 0x7fd76c0f0330 con 0x7fd778104920 2024-03-16T20:45:00.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.854+0000 7fd777fff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 0x7fd7600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:00.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.854+0000 7fd777fff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 0x7fd7600a3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd7680058c0 tx=0x7fd768005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:45:00.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:00.857+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd76c0b91e0 con 0x7fd778104920 2024-03-16T20:45:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.040+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd77810cf00 con 0x7fd778104920 2024-03-16T20:45:01.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.730+0000 7fd7757fa640 1 -- 172.21.15.23:0/169330393 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd76c0bc8a0 con 0x7fd778104920 2024-03-16T20:45:01.734 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2024-03-16T20:45:01.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 msgr2=0x7fd7600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 0x7fd7600a3ef0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd7680058c0 tx=0x7fd768005850 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 msgr2=0x7fd778197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c004270 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 shutdown_connections 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7fd7600a1a30 0x7fd7600a3ef0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd778107250 0x7fd77819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd778104920 0x7fd778197b90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 --2- 172.21.15.23:0/169330393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd778101ff0 0x7fd778197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 >> 172.21.15.23:0/169330393 conn(0x7fd7780fb980 msgr2=0x7fd778102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 shutdown_connections 2024-03-16T20:45:01.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.733+0000 7fd77e4ff640 1 -- 172.21.15.23:0/169330393 wait complete. 2024-03-16T20:45:01.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 -- 172.21.15.23:0/2594687924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 msgr2=0x7f9fb01043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/2594687924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb01043c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009ca0 tx=0x7f9fac02c0f0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 -- 172.21.15.23:0/2594687924 shutdown_connections 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/2594687924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb0109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/2594687924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/2594687924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb01043c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 -- 172.21.15.23:0/2594687924 >> 172.21.15.23:0/2594687924 conn(0x7f9fb00fb980 msgr2=0x7f9fb00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 -- 172.21.15.23:0/2594687924 shutdown_connections 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.864+0000 7f9fb86ff640 1 -- 172.21.15.23:0/2594687924 wait complete. 2024-03-16T20:45:01.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 Processor -- start 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 -- start start 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb019f060 con 0x7f9fb0101fd0 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb019f1d0 con 0x7f9fb0104900 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb86ff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb019f390 con 0x7f9fb0107230 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46478/0 (socket says 172.21.15.23:46478) 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 -- 172.21.15.23:0/3264915249 learned_addr learned my addr 172.21.15.23:0/3264915249 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb6c9c640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:01.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 -- 172.21.15.23:0/3264915249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 msgr2=0x7f9fb019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb5c9a640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 -- 172.21.15.23:0/3264915249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 msgr2=0x7f9fb0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 -- 172.21.15.23:0/3264915249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac009900 con 0x7f9fb0101fd0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb6c9c640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb5c9a640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fb649b640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fac032710 tx=0x7f9fac034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.865+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac003970 con 0x7f9fb0101fd0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.866+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fac004040 con 0x7f9fb0101fd0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.866+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb019f610 con 0x7f9fb0101fd0 2024-03-16T20:45:01.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.866+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0406b0 con 0x7f9fb0101fd0 2024-03-16T20:45:01.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.866+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb019fae0 con 0x7f9fb0101fd0 2024-03-16T20:45:01.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.867+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb00fd470 con 0x7f9fb0101fd0 2024-03-16T20:45:01.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.868+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 20) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9fac00a040 con 0x7f9fb0101fd0 2024-03-16T20:45:01.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.868+0000 7f9fa77fe640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 0x7f9f9c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-16T20:45:01.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.868+0000 7f9fb5c9a640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 0x7f9f9c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-16T20:45:01.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.868+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8496+0+0 (secure 0 0 0) 0x7f9fac0f09e0 con 0x7f9fb0101fd0 2024-03-16T20:45:01.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.869+0000 7f9fb5c9a640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 0x7f9f9c0a3f70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0198a80 tx=0x7f9fa0005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-16T20:45:01.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:01.872+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9fac0b9af0 con 0x7f9fb0101fd0 2024-03-16T20:45:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.054+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fb010cf00 con 0x7f9fb0101fd0 2024-03-16T20:45:02.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:01 smithi023 ceph-mon[31879]: pgmap v628: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-16T20:45:02.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:02.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:01 smithi023 ceph-mon[31879]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T20:45:02.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:01 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:02.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:01 smithi031 ceph-mon[36281]: pgmap v628: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-16T20:45:02.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:02.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:01 smithi031 ceph-mon[36281]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T20:45:02.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:01 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:02.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:01 smithi189 ceph-mon[36655]: pgmap v628: 129 pgs: 129 active+clean; 67 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-16T20:45:02.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/4082696219' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:02.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:01 smithi189 ceph-mon[36655]: osdmap e94: 12 total, 12 up, 12 in 2024-03-16T20:45:02.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:01 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:02.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.737+0000 7f9fa77fe640 1 -- 172.21.15.23:0/3264915249 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v96) v1 ==== 178+0+0 (secure 0 0 0) 0x7f9fac0041b0 con 0x7f9fb0101fd0 2024-03-16T20:45:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' removed 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 msgr2=0x7f9f9c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 0x7f9f9c0a3f70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0198a80 tx=0x7f9fa0005d90 comp rx=0 tx=0).stop 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 msgr2=0x7f9fb0197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fac032710 tx=0x7f9fac034cc0 comp rx=0 tx=0).stop 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 shutdown_connections 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:6800/3040096369,v1:172.21.15.23:6801/3040096369] conn(0x7f9f9c0a1ab0 0x7f9f9c0a3f70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fb0107230 0x7f9fb019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fb0104900 0x7f9fb0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:02.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 --2- 172.21.15.23:0/3264915249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb0101fd0 0x7f9fb0197570 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-16T20:45:02.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 >> 172.21.15.23:0/3264915249 conn(0x7f9fb00fb980 msgr2=0x7f9fb0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-16T20:45:02.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 shutdown_connections 2024-03-16T20:45:02.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-16T20:45:02.739+0000 7f9fb86ff640 1 -- 172.21.15.23:0/3264915249 wait complete. 2024-03-16T20:45:02.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:45:02.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:45:02.868 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:45:02.910 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T20:45:02.911 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T20:45:02.954 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T20:45:02.969 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T20:45:02.983 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T20:45:02.983 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T20:45:02.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a 2024-03-16T20:45:03.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:03.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:02 smithi023 ceph-mon[31879]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T20:45:03.020 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:02 smithi023 ceph-mon[31879]: from='client.? 172.21.15.23:0/3264915249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:02 smithi031 ceph-mon[36281]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T20:45:03.187 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:02 smithi031 ceph-mon[36281]: from='client.? 172.21.15.23:0/3264915249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/169330393' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-16T20:45:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:02 smithi189 ceph-mon[36655]: osdmap e95: 12 total, 12 up, 12 in 2024-03-16T20:45:03.220 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:02 smithi189 ceph-mon[36655]: from='client.? 172.21.15.23:0/3264915249' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-16T20:45:03.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 systemd[1]: Stopping Ceph mon.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:03.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a[31875]: 2024-03-16T20:45:03.354+0000 7f9a5e142700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:03.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a[31875]: 2024-03-16T20:45:03.354+0000 7f9a5e142700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T20:45:03.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 podman[89641]: 2024-03-16 20:45:03.540178446 +0000 UTC m=+0.291765801 container died bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T20:45:04.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 podman[89641]: 2024-03-16 20:45:03.963034481 +0000 UTC m=+0.714621835 container cleanup bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:45:04.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:03 smithi023 bash[89641]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a 2024-03-16T20:45:04.526 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 16 20:45:04 smithi023 podman[89652]: 2024-03-16 20:45:04.222755586 +0000 UTC m=+0.674079560 container remove bb9eecf4146985434589ef570bfef19dbc097b1340f8efde7b186e8d779a44a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-16T20:45:04.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T20:45:04.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:04.641 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T20:45:04.641 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-16T20:45:04.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.b 2024-03-16T20:45:04.937 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:04 smithi031 systemd[1]: Stopping Ceph mon.b for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b[36277]: 2024-03-16T20:45:05.026+0000 7fc232591700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b[36277]: 2024-03-16T20:45:05.026+0000 7fc232591700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T20:45:05.437 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 podman[70492]: 2024-03-16 20:45:05.210567484 +0000 UTC m=+0.299463940 container died cc4ab3c979c44a288e56fc1cdf5994bd418b40caf554298402c5dcb84b187feb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:45:05.764 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 podman[70492]: 2024-03-16 20:45:05.627560898 +0000 UTC m=+0.716457349 container cleanup cc4ab3c979c44a288e56fc1cdf5994bd418b40caf554298402c5dcb84b187feb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T20:45:05.764 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 bash[70492]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b 2024-03-16T20:45:06.015 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:05 smithi031 podman[70505]: 2024-03-16 20:45:05.896201047 +0000 UTC m=+0.674866473 container remove cc4ab3c979c44a288e56fc1cdf5994bd418b40caf554298402c5dcb84b187feb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-16T20:45:06.015 INFO:journalctl@ceph.mon.b.smithi031.stdout:Mar 16 20:45:06 smithi031 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-16T20:45:06.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T20:45:06.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:06.089 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-16T20:45:06.089 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-16T20:45:06.089 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.c 2024-03-16T20:45:06.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:06 smithi189 systemd[1]: Stopping Ceph mon.c for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:06.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:06 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c[36638]: 2024-03-16T20:45:06.471+0000 7f7fd2b1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:06.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:06 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c[36638]: 2024-03-16T20:45:06.471+0000 7f7fd2b1e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T20:45:07.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:06 smithi189 podman[71797]: 2024-03-16 20:45:06.726788604 +0000 UTC m=+0.385263386 container died 11bcfd600bc11020d9a274576b361e414bff4d4c1f6ae381249cc4ab68da5ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307) 2024-03-16T20:45:07.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 podman[71797]: 2024-03-16 20:45:07.173585726 +0000 UTC m=+0.832060497 container cleanup 11bcfd600bc11020d9a274576b361e414bff4d4c1f6ae381249cc4ab68da5ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T20:45:07.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 bash[71797]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c 2024-03-16T20:45:07.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T20:45:07.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 podman[71812]: 2024-03-16 20:45:07.400652185 +0000 UTC m=+0.674361113 container remove 11bcfd600bc11020d9a274576b361e414bff4d4c1f6ae381249cc4ab68da5ece (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T20:45:07.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-16T20:45:07.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 systemd[1]: Stopped Ceph mon.c for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:45:07.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 16 20:45:07 smithi189 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mon.c.service: Consumed 11.606s CPU time. 2024-03-16T20:45:07.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:07.715 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-16T20:45:07.715 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T20:45:07.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x 2024-03-16T20:45:08.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:07 smithi023 systemd[1]: Stopping Ceph mgr.x for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:08.571 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:08 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:45:08.266+0000 7f3d948ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:08.572 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:08 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x[32122]: 2024-03-16T20:45:08.266+0000 7f3d948ac700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T20:45:08.572 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:08 smithi023 podman[89757]: 2024-03-16 20:45:08.298402141 +0000 UTC m=+0.385686373 container died 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T20:45:08.838 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:08 smithi023 podman[89757]: 2024-03-16 20:45:08.675899572 +0000 UTC m=+0.763183804 container cleanup 3e3fa79e76d011fd4418e06fc283333924363598e75cbc65d61b3c27b2b20970 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:aba3c1d240c873fc0cd3e1d30cd86b2dbb71f24b, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=) 2024-03-16T20:45:08.839 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 16 20:45:08 smithi023 bash[89757]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-x 2024-03-16T20:45:09.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T20:45:09.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:09.120 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T20:45:09.121 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T20:45:09.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.y 2024-03-16T20:45:09.437 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:09 smithi031 systemd[1]: Stopping Ceph mgr.y for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:09.786 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:09 smithi031 podman[70621]: 2024-03-16 20:45:09.453752128 +0000 UTC m=+0.136652316 container died a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T20:45:09.786 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:09 smithi031 podman[70621]: 2024-03-16 20:45:09.783415213 +0000 UTC m=+0.466315397 container cleanup a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-16T20:45:10.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:09 smithi031 bash[70621]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y 2024-03-16T20:45:10.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:10 smithi031 podman[70635]: 2024-03-16 20:45:10.002029302 +0000 UTC m=+0.545759944 container remove a18bd07d2401dc54ba2dc0f5530b375310dce150dd8ac4de82eba7b8362f5a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:10.187 INFO:journalctl@ceph.mgr.y.smithi031.stdout:Mar 16 20:45:10 smithi031 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-16T20:45:10.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T20:45:10.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:10.303 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T20:45:10.303 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-16T20:45:10.303 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.z 2024-03-16T20:45:10.653 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:45:10 smithi189 systemd[1]: Stopping Ceph mgr.z for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:10.970 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:45:10 smithi189 podman[71929]: 2024-03-16 20:45:10.650051896 +0000 UTC m=+0.158166399 container died 89a5d412cebec62b1eb016f17e8ca6a6be23b7128bcc4659bb3b76ce1b0028bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-16T20:45:11.310 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:45:11 smithi189 podman[71929]: 2024-03-16 20:45:11.046795915 +0000 UTC m=+0.554910417 container cleanup 89a5d412cebec62b1eb016f17e8ca6a6be23b7128bcc4659bb3b76ce1b0028bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-16T20:45:11.311 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 16 20:45:11 smithi189 bash[71929]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mgr-z 2024-03-16T20:45:11.430 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-16T20:45:11.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:11.467 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-16T20:45:11.468 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T20:45:11.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.0 2024-03-16T20:45:11.817 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:11 smithi023 systemd[1]: Stopping Ceph osd.0 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:12.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:11 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0[43983]: 2024-03-16T20:45:11.813+0000 7f88e197c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:12.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:11 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0[43983]: 2024-03-16T20:45:11.813+0000 7f88e197c700 -1 osd.0 96 *** Got signal Terminated *** 2024-03-16T20:45:12.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:11 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0[43983]: 2024-03-16T20:45:11.813+0000 7f88e197c700 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:45:17.710 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:17 smithi023 podman[89878]: 2024-03-16 20:45:17.407882634 +0000 UTC m=+5.741759624 container died 0d1a9eaf3c7266aaabe72a875ba9c5d3e3ed02cf71d168ea16dd1cc764c2ae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T20:45:18.076 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:17 smithi023 podman[89878]: 2024-03-16 20:45:17.80563669 +0000 UTC m=+6.139513722 container cleanup 0d1a9eaf3c7266aaabe72a875ba9c5d3e3ed02cf71d168ea16dd1cc764c2ae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:18.076 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:17 smithi023 bash[89878]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0 2024-03-16T20:45:18.338 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89895]: 2024-03-16 20:45:18.073888572 +0000 UTC m=+0.655731288 container remove 0d1a9eaf3c7266aaabe72a875ba9c5d3e3ed02cf71d168ea16dd1cc764c2ae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-16T20:45:18.714 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89986]: 2024-03-16 20:45:18.320902133 +0000 UTC m=+0.017990851 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:45:18.714 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89986]: 2024-03-16 20:45:18.458254628 +0000 UTC m=+0.155343327 container create 08feb50b149c2e26a1c28b4286fc68526e08c8cb94408b8d836c7ef58c7f6a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-16T20:45:19.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89986]: 2024-03-16 20:45:18.71142997 +0000 UTC m=+0.408518679 container init 08feb50b149c2e26a1c28b4286fc68526e08c8cb94408b8d836c7ef58c7f6a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T20:45:19.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89986]: 2024-03-16 20:45:18.714976504 +0000 UTC m=+0.412065204 container start 08feb50b149c2e26a1c28b4286fc68526e08c8cb94408b8d836c7ef58c7f6a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T20:45:19.089 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:18 smithi023 podman[89986]: 2024-03-16 20:45:18.877624848 +0000 UTC m=+0.574713563 container attach 08feb50b149c2e26a1c28b4286fc68526e08c8cb94408b8d836c7ef58c7f6a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:45:20.683 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 16 20:45:20 smithi023 podman[89986]: 2024-03-16 20:45:20.37064575 +0000 UTC m=+2.067734452 container died 08feb50b149c2e26a1c28b4286fc68526e08c8cb94408b8d836c7ef58c7f6a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD) 2024-03-16T20:45:20.962 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T20:45:21.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:21.010 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T20:45:21.011 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T20:45:21.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.1 2024-03-16T20:45:21.333 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:21 smithi031 systemd[1]: Stopping Ceph osd.1 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:21.687 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:21 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:45:21.329+0000 7f3823e48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:21.687 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:21 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:45:21.329+0000 7f3823e48700 -1 osd.1 96 *** Got signal Terminated *** 2024-03-16T20:45:21.687 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:21 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1[40877]: 2024-03-16T20:45:21.329+0000 7f3823e48700 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:45:27.167 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:26 smithi031 podman[70745]: 2024-03-16 20:45:26.860607271 +0000 UTC m=+5.650581521 container died c0a73288a97604ad699af20c3a8cad0ca1a1800c823c46a3a35bc9612fe60c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T20:45:27.437 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:27 smithi031 podman[70745]: 2024-03-16 20:45:27.24624011 +0000 UTC m=+6.036214352 container cleanup c0a73288a97604ad699af20c3a8cad0ca1a1800c823c46a3a35bc9612fe60c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:27.437 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:27 smithi031 bash[70745]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1 2024-03-16T20:45:27.855 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:27 smithi031 podman[70764]: 2024-03-16 20:45:27.531197609 +0000 UTC m=+0.666717832 container remove c0a73288a97604ad699af20c3a8cad0ca1a1800c823c46a3a35bc9612fe60c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:28.106 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:27 smithi031 podman[70856]: 2024-03-16 20:45:27.754970789 +0000 UTC m=+0.019311731 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:45:28.106 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:27 smithi031 podman[70856]: 2024-03-16 20:45:27.990565653 +0000 UTC m=+0.254906576 container create 1ad07258fd58adaa478777abef10aaf3496971627da5a94d2967abcb44c7f1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:28.437 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:28 smithi031 podman[70856]: 2024-03-16 20:45:28.218387521 +0000 UTC m=+0.482728471 container init 1ad07258fd58adaa478777abef10aaf3496971627da5a94d2967abcb44c7f1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-16T20:45:28.437 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:28 smithi031 podman[70856]: 2024-03-16 20:45:28.221724234 +0000 UTC m=+0.486065160 container start 1ad07258fd58adaa478777abef10aaf3496971627da5a94d2967abcb44c7f1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T20:45:28.437 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:28 smithi031 podman[70856]: 2024-03-16 20:45:28.318388811 +0000 UTC m=+0.582729757 container attach 1ad07258fd58adaa478777abef10aaf3496971627da5a94d2967abcb44c7f1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T20:45:29.083 INFO:journalctl@ceph.osd.1.smithi031.stdout:Mar 16 20:45:28 smithi031 podman[70856]: 2024-03-16 20:45:28.712905945 +0000 UTC m=+0.977246876 container died 1ad07258fd58adaa478777abef10aaf3496971627da5a94d2967abcb44c7f1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T20:45:29.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T20:45:29.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:29.445 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T20:45:29.446 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T20:45:29.446 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.2 2024-03-16T20:45:29.750 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:29 smithi189 systemd[1]: Stopping Ceph osd.2 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:30.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:29 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2[41781]: 2024-03-16T20:45:29.745+0000 7efda5078700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:30.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:29 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2[41781]: 2024-03-16T20:45:29.745+0000 7efda5078700 -1 osd.2 96 *** Got signal Terminated *** 2024-03-16T20:45:30.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:29 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2[41781]: 2024-03-16T20:45:29.745+0000 7efda5078700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:45:35.558 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:35 smithi189 podman[72052]: 2024-03-16 20:45:35.260322487 +0000 UTC m=+5.629671621 container died 2a2abcaa72c99ec390bd4799814ea2c66127a7fae9cfc0d74b9b9dba5fc7a1d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:45:35.916 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:35 smithi189 podman[72052]: 2024-03-16 20:45:35.637514333 +0000 UTC m=+6.006863460 container cleanup 2a2abcaa72c99ec390bd4799814ea2c66127a7fae9cfc0d74b9b9dba5fc7a1d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-16T20:45:35.917 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:35 smithi189 bash[72052]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2 2024-03-16T20:45:36.220 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:35 smithi189 podman[72071]: 2024-03-16 20:45:35.913771284 +0000 UTC m=+0.649193646 container remove 2a2abcaa72c99ec390bd4799814ea2c66127a7fae9cfc0d74b9b9dba5fc7a1d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T20:45:36.512 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:36 smithi189 podman[72159]: 2024-03-16 20:45:36.145842192 +0000 UTC m=+0.018046407 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:45:36.512 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:36 smithi189 podman[72159]: 2024-03-16 20:45:36.264786682 +0000 UTC m=+0.136990889 container create a8b0a9925042f01221b29342b6de0132f7180706b14f5c349976bbcf7e17c7dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-16T20:45:36.512 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:36 smithi189 podman[72159]: 2024-03-16 20:45:36.509476291 +0000 UTC m=+0.381680509 container init a8b0a9925042f01221b29342b6de0132f7180706b14f5c349976bbcf7e17c7dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T20:45:36.970 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:36 smithi189 podman[72159]: 2024-03-16 20:45:36.513189349 +0000 UTC m=+0.385393557 container start a8b0a9925042f01221b29342b6de0132f7180706b14f5c349976bbcf7e17c7dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-16T20:45:36.970 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:36 smithi189 podman[72159]: 2024-03-16 20:45:36.651431219 +0000 UTC m=+0.523635486 container attach a8b0a9925042f01221b29342b6de0132f7180706b14f5c349976bbcf7e17c7dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T20:45:37.445 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 16 20:45:37 smithi189 podman[72159]: 2024-03-16 20:45:37.166663939 +0000 UTC m=+1.038868141 container died a8b0a9925042f01221b29342b6de0132f7180706b14f5c349976bbcf7e17c7dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-16T20:45:37.762 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T20:45:37.804 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:37.804 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T20:45:37.804 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T20:45:37.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.3 2024-03-16T20:45:38.089 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:37 smithi023 systemd[1]: Stopping Ceph osd.3 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:38.589 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:38 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:45:38.256+0000 7f43a50e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:38.589 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:38 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:45:38.256+0000 7f43a50e4700 -1 osd.3 96 *** Got signal Terminated *** 2024-03-16T20:45:38.589 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:38 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3[50958]: 2024-03-16T20:45:38.256+0000 7f43a50e4700 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:45:44.243 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:43 smithi023 podman[90093]: 2024-03-16 20:45:43.942144687 +0000 UTC m=+5.939532409 container died a6c2e637c50590a30c79f39d51acc0a6791a2b4fe196a72eb98f5c12f23ef79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-16T20:45:44.587 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:44 smithi023 podman[90093]: 2024-03-16 20:45:44.324225117 +0000 UTC m=+6.321612835 container cleanup a6c2e637c50590a30c79f39d51acc0a6791a2b4fe196a72eb98f5c12f23ef79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T20:45:44.587 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:44 smithi023 bash[90093]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3 2024-03-16T20:45:44.839 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:44 smithi023 podman[90122]: 2024-03-16 20:45:44.583887916 +0000 UTC m=+0.631159557 container remove a6c2e637c50590a30c79f39d51acc0a6791a2b4fe196a72eb98f5c12f23ef79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:45:45.224 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:44 smithi023 podman[90211]: 2024-03-16 20:45:44.802349775 +0000 UTC m=+0.018706822 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:45:45.224 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:44 smithi023 podman[90211]: 2024-03-16 20:45:44.901510592 +0000 UTC m=+0.117867683 container create 169d8da62071b6d87bb1419822a2eaecccc491ffc62c3d111fc456e8d36a6341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-16T20:45:45.224 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:45 smithi023 podman[90211]: 2024-03-16 20:45:45.129144553 +0000 UTC m=+0.345501612 container init 169d8da62071b6d87bb1419822a2eaecccc491ffc62c3d111fc456e8d36a6341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T20:45:45.224 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:45 smithi023 podman[90211]: 2024-03-16 20:45:45.132296955 +0000 UTC m=+0.348654009 container start 169d8da62071b6d87bb1419822a2eaecccc491ffc62c3d111fc456e8d36a6341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-03-16T20:45:45.224 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:45 smithi023 podman[90211]: 2024-03-16 20:45:45.220800164 +0000 UTC m=+0.437157219 container attach 169d8da62071b6d87bb1419822a2eaecccc491ffc62c3d111fc456e8d36a6341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-16T20:45:45.589 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 16 20:45:45 smithi023 podman[90211]: 2024-03-16 20:45:45.426356129 +0000 UTC m=+0.642713173 container died 169d8da62071b6d87bb1419822a2eaecccc491ffc62c3d111fc456e8d36a6341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T20:45:46.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T20:45:46.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:46.254 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T20:45:46.254 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T20:45:46.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.4 2024-03-16T20:45:46.571 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:46 smithi031 systemd[1]: Stopping Ceph osd.4 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:46.938 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:46 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4[47392]: 2024-03-16T20:45:46.567+0000 7fe4b5c3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:46.938 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:46 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4[47392]: 2024-03-16T20:45:46.567+0000 7fe4b5c3f700 -1 osd.4 96 *** Got signal Terminated *** 2024-03-16T20:45:46.939 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:46 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4[47392]: 2024-03-16T20:45:46.567+0000 7fe4b5c3f700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:45:52.437 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:52 smithi031 podman[70980]: 2024-03-16 20:45:52.166371325 +0000 UTC m=+5.711439986 container died 6b03886c1126bd79cc84a91d1bdcefdc1715e5214d4ce3e9f24bafbb478815bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-16T20:45:52.830 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:52 smithi031 podman[70980]: 2024-03-16 20:45:52.550569496 +0000 UTC m=+6.095638164 container cleanup 6b03886c1126bd79cc84a91d1bdcefdc1715e5214d4ce3e9f24bafbb478815bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:45:52.830 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:52 smithi031 bash[70980]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4 2024-03-16T20:45:53.156 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:52 smithi031 podman[70998]: 2024-03-16 20:45:52.827309009 +0000 UTC m=+0.659183071 container remove 6b03886c1126bd79cc84a91d1bdcefdc1715e5214d4ce3e9f24bafbb478815bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T20:45:53.418 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.153155213 +0000 UTC m=+0.103158066 container create 6a62b72ae3a432163a884f3f12a109086933fd601eb2e84b976b1f46d84d4309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T20:45:53.418 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.068159643 +0000 UTC m=+0.018162518 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:45:53.836 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.523200504 +0000 UTC m=+0.473203375 container init 6a62b72ae3a432163a884f3f12a109086933fd601eb2e84b976b1f46d84d4309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T20:45:53.836 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.526606948 +0000 UTC m=+0.476609807 container start 6a62b72ae3a432163a884f3f12a109086933fd601eb2e84b976b1f46d84d4309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-16T20:45:53.837 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.614206548 +0000 UTC m=+0.564209424 container attach 6a62b72ae3a432163a884f3f12a109086933fd601eb2e84b976b1f46d84d4309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:45:54.162 INFO:journalctl@ceph.osd.4.smithi031.stdout:Mar 16 20:45:53 smithi031 podman[71086]: 2024-03-16 20:45:53.833756151 +0000 UTC m=+0.783759013 container died 6a62b72ae3a432163a884f3f12a109086933fd601eb2e84b976b1f46d84d4309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:45:54.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T20:45:54.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:45:54.495 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T20:45:54.495 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T20:45:54.495 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.5 2024-03-16T20:45:54.821 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:45:54 smithi189 systemd[1]: Stopping Ceph osd.5 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:45:55.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:45:54 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5[48588]: 2024-03-16T20:45:54.816+0000 7f8bc95c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:45:55.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:45:54 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5[48588]: 2024-03-16T20:45:54.816+0000 7f8bc95c3700 -1 osd.5 96 *** Got signal Terminated *** 2024-03-16T20:45:55.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:45:54 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5[48588]: 2024-03-16T20:45:54.816+0000 7f8bc95c3700 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:00.896 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:00 smithi189 podman[72275]: 2024-03-16 20:46:00.604169348 +0000 UTC m=+5.912167907 container died cdc4ee51818863dd1e8014dc29deefa33ae22315fe311085397433481d2bb945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:46:01.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:00 smithi189 podman[72275]: 2024-03-16 20:46:00.95863959 +0000 UTC m=+6.266638141 container cleanup cdc4ee51818863dd1e8014dc29deefa33ae22315fe311085397433481d2bb945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T20:46:01.220 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:00 smithi189 bash[72275]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5 2024-03-16T20:46:01.599 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72292]: 2024-03-16 20:46:01.277097496 +0000 UTC m=+0.675226989 container remove cdc4ee51818863dd1e8014dc29deefa33ae22315fe311085397433481d2bb945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T20:46:01.885 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72386]: 2024-03-16 20:46:01.499666561 +0000 UTC m=+0.017668680 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:01.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72386]: 2024-03-16 20:46:01.62800035 +0000 UTC m=+0.146002471 container create 302ff6fea8a1b0b8db1eb5dc0018c78aed629b5263cfcf9952157a0e3a5d3ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T20:46:01.886 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72386]: 2024-03-16 20:46:01.88220831 +0000 UTC m=+0.400210443 container init 302ff6fea8a1b0b8db1eb5dc0018c78aed629b5263cfcf9952157a0e3a5d3ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T20:46:02.178 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72386]: 2024-03-16 20:46:01.885405892 +0000 UTC m=+0.403408014 container start 302ff6fea8a1b0b8db1eb5dc0018c78aed629b5263cfcf9952157a0e3a5d3ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:02.179 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:01 smithi189 podman[72386]: 2024-03-16 20:46:01.948756425 +0000 UTC m=+0.466758560 container attach 302ff6fea8a1b0b8db1eb5dc0018c78aed629b5263cfcf9952157a0e3a5d3ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-16T20:46:02.470 INFO:journalctl@ceph.osd.5.smithi189.stdout:Mar 16 20:46:02 smithi189 podman[72386]: 2024-03-16 20:46:02.176151579 +0000 UTC m=+0.694153720 container died 302ff6fea8a1b0b8db1eb5dc0018c78aed629b5263cfcf9952157a0e3a5d3ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307) 2024-03-16T20:46:02.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T20:46:02.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:02.835 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T20:46:02.835 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T20:46:02.836 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.6 2024-03-16T20:46:03.144 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:02 smithi023 systemd[1]: Stopping Ceph osd.6 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:03.589 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6[57485]: 2024-03-16T20:46:03.141+0000 7fc525b04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:03.589 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6[57485]: 2024-03-16T20:46:03.141+0000 7fc525b04700 -1 osd.6 96 *** Got signal Terminated *** 2024-03-16T20:46:03.589 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:03 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6[57485]: 2024-03-16T20:46:03.141+0000 7fc525b04700 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:09.103 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:08 smithi023 podman[90317]: 2024-03-16 20:46:08.84613488 +0000 UTC m=+5.820986626 container died e5e25ad3dec2c1f204d0e4eae4478c5c05b60be4bd63d2febe0c71b21a2dc760 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-16T20:46:09.442 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:09 smithi023 podman[90317]: 2024-03-16 20:46:09.191147799 +0000 UTC m=+6.165999553 container cleanup e5e25ad3dec2c1f204d0e4eae4478c5c05b60be4bd63d2febe0c71b21a2dc760 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T20:46:09.442 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:09 smithi023 bash[90317]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6 2024-03-16T20:46:09.779 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:09 smithi023 podman[90334]: 2024-03-16 20:46:09.463739579 +0000 UTC m=+0.622154628 container remove e5e25ad3dec2c1f204d0e4eae4478c5c05b60be4bd63d2febe0c71b21a2dc760 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:10.089 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:09 smithi023 podman[90432]: 2024-03-16 20:46:09.776964963 +0000 UTC m=+0.116378732 container create 5601950122cf7d2cd633d1a8bbfb3337548c6b6c82dc90ef11a47d8d0d354700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T20:46:10.089 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:09 smithi023 podman[90432]: 2024-03-16 20:46:09.684616028 +0000 UTC m=+0.024029807 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:10.089 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:10 smithi023 podman[90432]: 2024-03-16 20:46:10.004718308 +0000 UTC m=+0.344132090 container init 5601950122cf7d2cd633d1a8bbfb3337548c6b6c82dc90ef11a47d8d0d354700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T20:46:10.089 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:10 smithi023 podman[90432]: 2024-03-16 20:46:10.008253001 +0000 UTC m=+0.347666777 container start 5601950122cf7d2cd633d1a8bbfb3337548c6b6c82dc90ef11a47d8d0d354700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:10.589 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:10 smithi023 podman[90432]: 2024-03-16 20:46:10.229563738 +0000 UTC m=+0.568977509 container attach 5601950122cf7d2cd633d1a8bbfb3337548c6b6c82dc90ef11a47d8d0d354700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-16T20:46:10.589 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 16 20:46:10 smithi023 podman[90432]: 2024-03-16 20:46:10.317539086 +0000 UTC m=+0.656952864 container died 5601950122cf7d2cd633d1a8bbfb3337548c6b6c82dc90ef11a47d8d0d354700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T20:46:10.878 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T20:46:10.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:10.919 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T20:46:10.920 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T20:46:10.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.7 2024-03-16T20:46:11.265 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:10 smithi031 systemd[1]: Stopping Ceph osd.7 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:11.549 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:11 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7[54399]: 2024-03-16T20:46:11.261+0000 7fadaa83f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:11.549 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:11 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7[54399]: 2024-03-16T20:46:11.261+0000 7fadaa83f700 -1 osd.7 96 *** Got signal Terminated *** 2024-03-16T20:46:11.549 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:11 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7[54399]: 2024-03-16T20:46:11.261+0000 7fadaa83f700 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:17.097 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:16 smithi031 podman[71192]: 2024-03-16 20:46:16.824833486 +0000 UTC m=+5.696779059 container died 71b5267411aa91dd1b93cd863145efb88a4824af707f57f9398e596ec9e92d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T20:46:17.097 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71192]: 2024-03-16 20:46:17.09454775 +0000 UTC m=+5.966493309 container cleanup 71b5267411aa91dd1b93cd863145efb88a4824af707f57f9398e596ec9e92d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-16T20:46:17.437 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 bash[71192]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7 2024-03-16T20:46:17.437 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71222]: 2024-03-16 20:46:17.296186175 +0000 UTC m=+0.470347958 container remove 71b5267411aa91dd1b93cd863145efb88a4824af707f57f9398e596ec9e92d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T20:46:18.044 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:17 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:17.949+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:18.044 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71311]: 2024-03-16 20:46:17.612481786 +0000 UTC m=+0.017803408 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:18.044 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71311]: 2024-03-16 20:46:17.713640371 +0000 UTC m=+0.118961975 container create 0b5fee0344bf635fd137e639c26b04e1bb611431410ba094fce77337abd75126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T20:46:18.044 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71311]: 2024-03-16 20:46:17.94149079 +0000 UTC m=+0.346812420 container init 0b5fee0344bf635fd137e639c26b04e1bb611431410ba094fce77337abd75126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-16T20:46:18.045 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:17 smithi031 podman[71311]: 2024-03-16 20:46:17.944633727 +0000 UTC m=+0.349955331 container start 0b5fee0344bf635fd137e639c26b04e1bb611431410ba094fce77337abd75126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-16T20:46:18.045 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:18 smithi031 podman[71311]: 2024-03-16 20:46:18.041311872 +0000 UTC m=+0.446633494 container attach 0b5fee0344bf635fd137e639c26b04e1bb611431410ba094fce77337abd75126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T20:46:18.437 INFO:journalctl@ceph.osd.7.smithi031.stdout:Mar 16 20:46:18 smithi031 podman[71311]: 2024-03-16 20:46:18.252779271 +0000 UTC m=+0.658100884 container died 0b5fee0344bf635fd137e639c26b04e1bb611431410ba094fce77337abd75126 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-16T20:46:18.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T20:46:18.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:18.825 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T20:46:18.825 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-16T20:46:18.825 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.8 2024-03-16T20:46:19.132 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:18 smithi189 systemd[1]: Stopping Ceph osd.8 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:19.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:18 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:18.908+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:19.402 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:19 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:19.190+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:19.403 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:19 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8[55476]: 2024-03-16T20:46:19.128+0000 7f41cdab4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:19.403 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:19 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8[55476]: 2024-03-16T20:46:19.128+0000 7f41cdab4700 -1 osd.8 96 *** Got signal Terminated *** 2024-03-16T20:46:19.403 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:19 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8[55476]: 2024-03-16T20:46:19.128+0000 7f41cdab4700 -1 osd.8 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:20.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:19 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:19.916+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:20.719 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:20 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:20.225+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:21.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:20 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:20.877+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:21.470 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:21 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:21.211+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:22.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:21 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:21.859+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:22.341 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:21 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:21.934+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:22.470 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:22 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:22.208+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:23.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:22 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:22.853+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:23.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:22 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:22.968+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:23.470 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:23 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:23.180+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:24.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:23 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:23.844+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:24.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:24 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:24.003+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:24.470 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:24 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:24.201+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:25.016 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:24 smithi189 podman[72491]: 2024-03-16 20:46:24.728483222 +0000 UTC m=+5.713440432 container died 945e4cfd0d882798486beba4d9bf322e566c0ce675aacfdfd4257bb8ec04ad09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T20:46:25.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:24 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:24.822+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:25.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:24 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:24.998+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:25.358 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:25 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:25.221+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:25.358 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72491]: 2024-03-16 20:46:25.104149926 +0000 UTC m=+6.089107137 container cleanup 945e4cfd0d882798486beba4d9bf322e566c0ce675aacfdfd4257bb8ec04ad09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:46:25.358 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 bash[72491]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8 2024-03-16T20:46:25.651 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72520]: 2024-03-16 20:46:25.355384029 +0000 UTC m=+0.625941233 container remove 945e4cfd0d882798486beba4d9bf322e566c0ce675aacfdfd4257bb8ec04ad09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T20:46:25.937 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72613]: 2024-03-16 20:46:25.551413504 +0000 UTC m=+0.017732875 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:25.937 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72613]: 2024-03-16 20:46:25.673075205 +0000 UTC m=+0.139394560 container create 9ad3397b6ca66b14a2e993384e2c99b1f0cdae943322056ece91db4068e66dca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T20:46:26.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:25 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:25.822+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:26.220 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72613]: 2024-03-16 20:46:25.933861682 +0000 UTC m=+0.400181024 container init 9ad3397b6ca66b14a2e993384e2c99b1f0cdae943322056ece91db4068e66dca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:26.220 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:25 smithi189 podman[72613]: 2024-03-16 20:46:25.937242502 +0000 UTC m=+0.403561830 container start 9ad3397b6ca66b14a2e993384e2c99b1f0cdae943322056ece91db4068e66dca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:46:26.220 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:26 smithi189 podman[72613]: 2024-03-16 20:46:26.042155486 +0000 UTC m=+0.508474847 container attach 9ad3397b6ca66b14a2e993384e2c99b1f0cdae943322056ece91db4068e66dca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:46:26.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:25 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:25.969+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:26.582 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:26 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:26.262+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:26.582 INFO:journalctl@ceph.osd.8.smithi189.stdout:Mar 16 20:46:26 smithi189 podman[72613]: 2024-03-16 20:46:26.228073921 +0000 UTC m=+0.694393284 container died 9ad3397b6ca66b14a2e993384e2c99b1f0cdae943322056ece91db4068e66dca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0) 2024-03-16T20:46:26.868 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.8.service' 2024-03-16T20:46:26.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:26.909 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-16T20:46:26.909 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-16T20:46:26.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.9 2024-03-16T20:46:27.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:26 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:26.772+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:27.213 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:26 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:26.920+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:27.213 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:26 smithi023 systemd[1]: Stopping Ceph osd.9 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:27.589 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:27 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:27.209+0000 7f761e7d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:27.589 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:27 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:27.209+0000 7f761e7d7700 -1 osd.9 96 *** Got signal Terminated *** 2024-03-16T20:46:27.589 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:27 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:27.209+0000 7f761e7d7700 -1 osd.9 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:27.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:27 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:27.304+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:28.096 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:27 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:27.727+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:28.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:27 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:27.896+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:28.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:28 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:28.292+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:29.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:28 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:28.693+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:29.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:28 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:28.907+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:29.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:29 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:29.289+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:30.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:29 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:29.689+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:30.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:29 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:29.689+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:30.338 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:29 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:29.899+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:30.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:29 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:29.899+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:25.380567+0000 front 2024-03-16T20:45:25.380741+0000 (oldest deadline 2024-03-16T20:46:29.480569+0000) 2024-03-16T20:46:30.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:30 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:30.319+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:30.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:30 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:30.653+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:30.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:30 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:30.653+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:31.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:30 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:30.851+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:31.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:30 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:30.851+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:25.380567+0000 front 2024-03-16T20:45:25.380741+0000 (oldest deadline 2024-03-16T20:46:29.480569+0000) 2024-03-16T20:46:31.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:31 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:31.322+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:31.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:31 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:31.675+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:31.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:31 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:31.675+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:32.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:31 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:31.842+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.577651+0000 front 2024-03-16T20:45:15.577704+0000 (oldest deadline 2024-03-16T20:46:21.378670+0000) 2024-03-16T20:46:32.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:31 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9[64070]: 2024-03-16T20:46:31.842+0000 7f7616dc3700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:25.380567+0000 front 2024-03-16T20:45:25.380741+0000 (oldest deadline 2024-03-16T20:46:29.480569+0000) 2024-03-16T20:46:32.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:32 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:32.273+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:32.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:32 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:32.651+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:32.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:32 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:32.651+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:33.235 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:32 smithi023 podman[90539]: 2024-03-16 20:46:32.96476668 +0000 UTC m=+5.851433988 container died 0a18d1091c2ada7b5002d3f7c4a9c05db62eccf98211fae30c34c972e654530d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T20:46:33.588 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:33 smithi023 podman[90539]: 2024-03-16 20:46:33.342756719 +0000 UTC m=+6.229424027 container cleanup 0a18d1091c2ada7b5002d3f7c4a9c05db62eccf98211fae30c34c972e654530d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T20:46:33.588 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:33 smithi023 bash[90539]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9 2024-03-16T20:46:33.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:33 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:33.262+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:33.838 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:33 smithi023 podman[90556]: 2024-03-16 20:46:33.585385384 +0000 UTC m=+0.615671387 container remove 0a18d1091c2ada7b5002d3f7c4a9c05db62eccf98211fae30c34c972e654530d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5) 2024-03-16T20:46:34.150 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:33 smithi023 podman[90648]: 2024-03-16 20:46:33.795668814 +0000 UTC m=+0.018767934 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:34.151 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:34 smithi023 podman[90648]: 2024-03-16 20:46:34.019746598 +0000 UTC m=+0.242845744 container create 668c365b52b1d1e96599470c12e038dbfee1491fa32cbdf4f8b8c93e163db635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:46:34.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:33 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:33.685+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:34.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:33 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:33.685+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:34.551 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:34 smithi023 podman[90648]: 2024-03-16 20:46:34.246986011 +0000 UTC m=+0.470085129 container init 668c365b52b1d1e96599470c12e038dbfee1491fa32cbdf4f8b8c93e163db635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True) 2024-03-16T20:46:34.551 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:34 smithi023 podman[90648]: 2024-03-16 20:46:34.25048951 +0000 UTC m=+0.473588614 container start 668c365b52b1d1e96599470c12e038dbfee1491fa32cbdf4f8b8c93e163db635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, ceph=True) 2024-03-16T20:46:34.551 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:34 smithi023 podman[90648]: 2024-03-16 20:46:34.3134347 +0000 UTC m=+0.536533811 container attach 668c365b52b1d1e96599470c12e038dbfee1491fa32cbdf4f8b8c93e163db635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, ceph=True) 2024-03-16T20:46:34.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:34 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:34.299+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:34.838 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 16 20:46:34 smithi023 podman[90648]: 2024-03-16 20:46:34.548937016 +0000 UTC m=+0.772036135 container died 668c365b52b1d1e96599470c12e038dbfee1491fa32cbdf4f8b8c93e163db635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T20:46:34.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:34 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:34.683+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:34.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:34 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:34.683+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:35.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.9.service' 2024-03-16T20:46:35.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:35.155 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-16T20:46:35.156 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-16T20:46:35.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.10 2024-03-16T20:46:35.467 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 systemd[1]: Stopping Ceph osd.10 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:35.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:35 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:35.288+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:35.721 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:35.462+0000 7f11c6663700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:35.721 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:35.462+0000 7f11c6663700 -1 osd.10 96 *** Got signal Terminated *** 2024-03-16T20:46:35.721 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:35.462+0000 7f11c6663700 -1 osd.10 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:35.721 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:35.716+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:35.721 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:35 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:35.716+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:36.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:36 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:36.316+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:37.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:36 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:36.756+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:37.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:36 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:36.756+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:37.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:37 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:37.274+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:38.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:37 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:37.719+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:38.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:37 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:37.720+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:38.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:37 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:37.720+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:32.532758+0000 front 2024-03-16T20:45:32.532972+0000 (oldest deadline 2024-03-16T20:46:37.232829+0000) 2024-03-16T20:46:38.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:38 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:38.305+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:38.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:38 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:38.305+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:39.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:38 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:38.690+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:39.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:38 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:38.690+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:39.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:38 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:38.690+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:32.532758+0000 front 2024-03-16T20:45:32.532972+0000 (oldest deadline 2024-03-16T20:46:37.232829+0000) 2024-03-16T20:46:39.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:39 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:39.340+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:39.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:39 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:39.340+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:39.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:39 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:39.647+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:15.828415+0000 front 2024-03-16T20:45:15.828496+0000 (oldest deadline 2024-03-16T20:46:17.528548+0000) 2024-03-16T20:46:39.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:39 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:39.647+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6804 osd.1 since back 2024-03-16T20:45:26.131498+0000 front 2024-03-16T20:45:26.131403+0000 (oldest deadline 2024-03-16T20:46:29.031514+0000) 2024-03-16T20:46:39.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:39 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10[61294]: 2024-03-16T20:46:39.647+0000 7f11c2c57700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:32.532758+0000 front 2024-03-16T20:45:32.532972+0000 (oldest deadline 2024-03-16T20:46:37.232829+0000) 2024-03-16T20:46:40.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:40 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:40.370+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:40.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:40 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:40.370+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:41.187 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:40 smithi031 podman[71417]: 2024-03-16 20:46:40.921189654 +0000 UTC m=+5.562066446 container died fb1cc96208b590e731197ba2902ac60668e4048abf72aae13d8813e687b4872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-16T20:46:41.650 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:41 smithi031 podman[71417]: 2024-03-16 20:46:41.37972492 +0000 UTC m=+6.020601708 container cleanup fb1cc96208b590e731197ba2902ac60668e4048abf72aae13d8813e687b4872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T20:46:41.650 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:41 smithi031 bash[71417]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10 2024-03-16T20:46:41.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:41 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:41.368+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:41.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:41 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:41.368+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:41.937 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:41 smithi031 podman[71435]: 2024-03-16 20:46:41.647472946 +0000 UTC m=+0.723354327 container remove fb1cc96208b590e731197ba2902ac60668e4048abf72aae13d8813e687b4872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:46:42.437 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:41.907468647 +0000 UTC m=+0.018024977 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:42.437 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:42.00666313 +0000 UTC m=+0.117219473 container create 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True) 2024-03-16T20:46:42.437 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:42.183840036 +0000 UTC m=+0.294396359 container init 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T20:46:42.437 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:42.187560632 +0000 UTC m=+0.298116941 container start 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T20:46:42.438 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:42.2504731 +0000 UTC m=+0.361029425 container attach 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-16T20:46:42.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:42 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:42.397+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:42.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:42 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:42.397+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:42.882 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:42 smithi031 podman[71526]: 2024-03-16 20:46:42.500692679 +0000 UTC m=+0.611248988 container died 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T20:46:43.134 INFO:journalctl@ceph.osd.10.smithi031.stdout:Mar 16 20:46:43 smithi031 podman[71553]: 2024-03-16 20:46:43.129330072 +0000 UTC m=+0.620086559 container remove 60ede3e72c1712e80220d4ed848b8d7a1647f4397e2924b4e347cd0cfea06f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:43.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.10.service' 2024-03-16T20:46:43.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:43.192 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-16T20:46:43.192 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-16T20:46:43.192 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.11 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 systemd[1]: Stopping Ceph osd.11 for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:43.421+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:43.421+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:43.513+0000 7f222c42f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:43.513+0000 7f222c42f700 -1 osd.11 96 *** Got signal Terminated *** 2024-03-16T20:46:43.518 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:43 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:43.513+0000 7f222c42f700 -1 osd.11 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T20:46:44.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:44 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:44.397+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:44.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:44 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:44.397+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:44.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:44 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:44.397+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-16T20:45:43.198515+0000 front 2024-03-16T20:45:43.198342+0000 (oldest deadline 2024-03-16T20:46:44.298200+0000) 2024-03-16T20:46:45.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:45 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:45.393+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:45.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:45 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:45.393+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:45.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:45 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:45.393+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-16T20:45:43.198515+0000 front 2024-03-16T20:45:43.198342+0000 (oldest deadline 2024-03-16T20:46:44.298200+0000) 2024-03-16T20:46:46.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:46 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:46.439+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:46.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:46 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:46.439+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:46.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:46 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:46.439+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-16T20:45:43.198515+0000 front 2024-03-16T20:45:43.198342+0000 (oldest deadline 2024-03-16T20:46:44.298200+0000) 2024-03-16T20:46:47.970 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:47 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:47.466+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:47.970 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:47 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:47.466+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:47.970 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:47 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:47.466+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-16T20:45:43.198515+0000 front 2024-03-16T20:45:43.198342+0000 (oldest deadline 2024-03-16T20:46:44.298200+0000) 2024-03-16T20:46:48.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:48 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:48.453+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-16T20:45:14.291771+0000 front 2024-03-16T20:45:14.291692+0000 (oldest deadline 2024-03-16T20:46:18.391829+0000) 2024-03-16T20:46:48.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:48 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:48.453+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-03-16T20:45:34.496596+0000 front 2024-03-16T20:45:34.496640+0000 (oldest deadline 2024-03-16T20:46:37.996716+0000) 2024-03-16T20:46:48.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:48 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11[62113]: 2024-03-16T20:46:48.453+0000 7f2224a1b700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-16T20:45:43.198515+0000 front 2024-03-16T20:45:43.198342+0000 (oldest deadline 2024-03-16T20:46:44.298200+0000) 2024-03-16T20:46:49.470 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:49 smithi189 podman[72717]: 2024-03-16 20:46:49.187894586 +0000 UTC m=+5.800762493 container died 0c312cb6bdcbe7b963a776f5b9f21f21073dcce514953df2ec89b30d929fd8df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-16T20:46:49.818 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:49 smithi189 podman[72717]: 2024-03-16 20:46:49.555812138 +0000 UTC m=+6.168680045 container cleanup 0c312cb6bdcbe7b963a776f5b9f21f21073dcce514953df2ec89b30d929fd8df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T20:46:49.818 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:49 smithi189 bash[72717]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11 2024-03-16T20:46:50.162 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:49 smithi189 podman[72736]: 2024-03-16 20:46:49.815447159 +0000 UTC m=+0.627426264 container remove 0c312cb6bdcbe7b963a776f5b9f21f21073dcce514953df2ec89b30d929fd8df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True) 2024-03-16T20:46:50.430 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.062123421 +0000 UTC m=+0.017748707 image pull 80ec0d3daaba7a52b05a2c1e9fce2fdb808f0661f4843706a56e0840fed82af0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3 2024-03-16T20:46:50.430 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.166230889 +0000 UTC m=+0.121856161 container create cf47e2a69cc77c9662a7c0b7ffe75ed27496d715d9692e062358d00a2c3b0e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T20:46:50.430 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.360726204 +0000 UTC m=+0.316351499 container init cf47e2a69cc77c9662a7c0b7ffe75ed27496d715d9692e062358d00a2c3b0e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-16T20:46:50.430 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.36414023 +0000 UTC m=+0.319765509 container start cf47e2a69cc77c9662a7c0b7ffe75ed27496d715d9692e062358d00a2c3b0e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T20:46:50.430 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.426714699 +0000 UTC m=+0.382339978 container attach cf47e2a69cc77c9662a7c0b7ffe75ed27496d715d9692e062358d00a2c3b0e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:46:50.720 INFO:journalctl@ceph.osd.11.smithi189.stdout:Mar 16 20:46:50 smithi189 podman[72829]: 2024-03-16 20:46:50.650284887 +0000 UTC m=+0.605910171 container died cf47e2a69cc77c9662a7c0b7ffe75ed27496d715d9692e062358d00a2c3b0e01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS) 2024-03-16T20:46:51.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@osd.11.service' 2024-03-16T20:46:51.283 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:51.284 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-16T20:46:51.284 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-16T20:46:51.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.a 2024-03-16T20:46:51.580 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:51 smithi023 systemd[1]: Stopping Ceph mds.a for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:51.839 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:51 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: 2024-03-16T20:46:51.576+0000 7f5394643700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:51.839 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:51 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a[70792]: 2024-03-16T20:46:51.576+0000 7f5394643700 -1 mds.a *** got signal Terminated *** 2024-03-16T20:46:55.220 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:54 smithi023 podman[90765]: 2024-03-16 20:46:54.837224414 +0000 UTC m=+3.356107901 container died 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T20:46:55.589 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:55 smithi023 podman[90765]: 2024-03-16 20:46:55.284543494 +0000 UTC m=+3.803426986 container cleanup 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-16T20:46:55.589 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:55 smithi023 bash[90765]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a 2024-03-16T20:46:55.589 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 16 20:46:55 smithi023 podman[90776]: 2024-03-16 20:46:55.418965802 +0000 UTC m=+0.576432449 container remove 313baabf837934f731d682e7ad5e59723fe80cde9b682c62fe262ed79aa8d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux ) 2024-03-16T20:46:55.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.a.service' 2024-03-16T20:46:55.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:55.683 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-16T20:46:55.683 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-16T20:46:55.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.d 2024-03-16T20:46:56.007 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:55 smithi023 systemd[1]: Stopping Ceph mds.d for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:56.338 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:56 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d[69831]: 2024-03-16T20:46:56.004+0000 7f31f7e8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:56.339 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:56 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d[69831]: 2024-03-16T20:46:56.004+0000 7f31f7e8c700 -1 mds.d *** got signal Terminated *** 2024-03-16T20:46:58.490 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 podman[90887]: 2024-03-16 20:46:58.223897119 +0000 UTC m=+2.321382894 container died 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:46:58.750 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 podman[90887]: 2024-03-16 20:46:58.563105029 +0000 UTC m=+2.660590817 container cleanup 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-16T20:46:58.750 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 bash[90887]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d 2024-03-16T20:46:58.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.d.service' 2024-03-16T20:46:59.010 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 podman[90900]: 2024-03-16 20:46:58.747476931 +0000 UTC m=+0.518783434 container remove 070cec8073250333e94ed16c23a36ad269916624c011ded7ee45224233aece35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-16T20:46:59.010 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.d.service: Deactivated successfully. 2024-03-16T20:46:59.010 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 systemd[1]: Stopped Ceph mds.d for 3d249ea8-e3d3-11ee-95c9-87774f69a715. 2024-03-16T20:46:59.010 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 16 20:46:58 smithi023 systemd[1]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.d.service: Consumed 1.024s CPU time. 2024-03-16T20:46:59.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:46:59.022 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-16T20:46:59.022 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-16T20:46:59.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.g 2024-03-16T20:46:59.336 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:46:59 smithi023 systemd[1]: Stopping Ceph mds.g for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:46:59.589 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:46:59 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: 2024-03-16T20:46:59.332+0000 7f6dbe35a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:46:59.589 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:46:59 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g[70436]: 2024-03-16T20:46:59.332+0000 7f6dbe35a700 -1 mds.g *** got signal Terminated *** 2024-03-16T20:47:03.089 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:47:02 smithi023 podman[91005]: 2024-03-16 20:47:02.827097452 +0000 UTC m=+3.591678956 container died 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-16T20:47:03.570 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:47:03 smithi023 podman[91005]: 2024-03-16 20:47:03.282729638 +0000 UTC m=+4.047311141 container cleanup 7fbb4133a62b870ccdebf030369e690687d616840d6a4c356f8fe50ac74622a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T20:47:03.570 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 16 20:47:03 smithi023 bash[91005]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-g 2024-03-16T20:47:03.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.g.service' 2024-03-16T20:47:03.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:03.752 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-16T20:47:03.752 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-16T20:47:03.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.j 2024-03-16T20:47:04.088 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:03 smithi023 systemd[1]: Stopping Ceph mds.j for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:04.089 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j[69466]: 2024-03-16T20:47:04.069+0000 7fdf2f525700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:04.089 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:04 smithi023 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j[69466]: 2024-03-16T20:47:04.069+0000 7fdf2f525700 -1 mds.j *** got signal Terminated *** 2024-03-16T20:47:07.806 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:07 smithi023 podman[91125]: 2024-03-16 20:47:07.495675488 +0000 UTC m=+3.524544322 container died a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T20:47:07.806 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:07 smithi023 podman[91125]: 2024-03-16 20:47:07.803255726 +0000 UTC m=+3.832124629 container cleanup a86939ed9d53129764f8d7465bcdaa52442267bd615e5d24a538077b5f2edb5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T20:47:08.089 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 16 20:47:07 smithi023 bash[91125]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-j 2024-03-16T20:47:08.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.j.service' 2024-03-16T20:47:08.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:08.348 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-16T20:47:08.348 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-16T20:47:08.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.b 2024-03-16T20:47:08.648 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:08 smithi031 systemd[1]: Stopping Ceph mds.b for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:08.937 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:08 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: 2024-03-16T20:47:08.645+0000 7fd50431c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:08.937 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:08 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b[66834]: 2024-03-16T20:47:08.645+0000 7fd50431c700 -1 mds.b *** got signal Terminated *** 2024-03-16T20:47:11.083 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:10 smithi031 podman[71631]: 2024-03-16 20:47:10.789984536 +0000 UTC m=+2.244804628 container died 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T20:47:11.437 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:11 smithi031 podman[71631]: 2024-03-16 20:47:11.170242519 +0000 UTC m=+2.625062619 container cleanup 3535f825bb2d35a88c0c16caa934939abb48d045e3b6b305cda5cccefe6fb847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD) 2024-03-16T20:47:11.437 INFO:journalctl@ceph.mds.b.smithi031.stdout:Mar 16 20:47:11 smithi031 bash[71631]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-b 2024-03-16T20:47:11.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.b.service' 2024-03-16T20:47:11.717 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:11.717 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-16T20:47:11.717 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-16T20:47:11.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.e 2024-03-16T20:47:11.937 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:11 smithi031 systemd[1]: Stopping Ceph mds.e for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:12.437 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:12 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: 2024-03-16T20:47:12.115+0000 7fd9fc95a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:12.437 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:12 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e[66201]: 2024-03-16T20:47:12.115+0000 7fd9fc95a700 -1 mds.e *** got signal Terminated *** 2024-03-16T20:47:15.127 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:14 smithi031 podman[71753]: 2024-03-16 20:47:14.795250715 +0000 UTC m=+2.855030858 container died 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, org.label-schema.build-date=20240307, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-16T20:47:15.127 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:15 smithi031 podman[71753]: 2024-03-16 20:47:15.123946437 +0000 UTC m=+3.183726566 container cleanup 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-16T20:47:15.437 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:15 smithi031 bash[71753]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e 2024-03-16T20:47:15.437 INFO:journalctl@ceph.mds.e.smithi031.stdout:Mar 16 20:47:15 smithi031 podman[71766]: 2024-03-16 20:47:15.316851152 +0000 UTC m=+0.519655104 container remove 8954f2554da9ba80851b77f6c3136cb14d4465ae4f1c0055a5ef2d1d15515930 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-squid-aba3c1d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T20:47:15.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.e.service' 2024-03-16T20:47:15.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:15.592 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-16T20:47:15.593 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-16T20:47:15.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.h 2024-03-16T20:47:15.906 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:15 smithi031 systemd[1]: Stopping Ceph mds.h for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:16.187 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:15 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: 2024-03-16T20:47:15.902+0000 7fad5302f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:16.187 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:15 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h[66511]: 2024-03-16T20:47:15.902+0000 7fad5302f700 -1 mds.h *** got signal Terminated *** 2024-03-16T20:47:19.113 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:18 smithi031 podman[71876]: 2024-03-16 20:47:18.797251974 +0000 UTC m=+2.974994462 container died 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d) 2024-03-16T20:47:19.113 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:19 smithi031 podman[71876]: 2024-03-16 20:47:19.110540307 +0000 UTC m=+3.288282804 container cleanup 85cc5cf23ed91f84b6ef72f879e55e0f2f794b03a69a87104fa0d678ccc33e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T20:47:19.437 INFO:journalctl@ceph.mds.h.smithi031.stdout:Mar 16 20:47:19 smithi031 bash[71876]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-h 2024-03-16T20:47:19.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.h.service' 2024-03-16T20:47:19.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:19.638 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-16T20:47:19.638 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-16T20:47:19.638 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.k 2024-03-16T20:47:19.967 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:19 smithi031 systemd[1]: Stopping Ceph mds.k for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:20.437 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:19 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: 2024-03-16T20:47:19.964+0000 7f8057732700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:20.437 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:19 smithi031 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k[65876]: 2024-03-16T20:47:19.964+0000 7f8057732700 -1 mds.k *** got signal Terminated *** 2024-03-16T20:47:23.185 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:22 smithi031 podman[71995]: 2024-03-16 20:47:22.791271873 +0000 UTC m=+2.928342125 container died 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:47:23.437 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:23 smithi031 podman[71995]: 2024-03-16 20:47:23.264222692 +0000 UTC m=+3.401292919 container cleanup 7842f3b7956ca7d7e4834dfbb70b81500f8fe12262f180ddf736b8db8db725d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T20:47:23.438 INFO:journalctl@ceph.mds.k.smithi031.stdout:Mar 16 20:47:23 smithi031 bash[71995]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-k 2024-03-16T20:47:23.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.k.service' 2024-03-16T20:47:23.686 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:23.687 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-16T20:47:23.687 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-16T20:47:23.687 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.c 2024-03-16T20:47:23.970 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:23 smithi189 systemd[1]: Stopping Ceph mds.c for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:24.470 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:23 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: 2024-03-16T20:47:23.996+0000 7f96a0b6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:24.470 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:23 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c[67479]: 2024-03-16T20:47:23.996+0000 7f96a0b6b700 -1 mds.c *** got signal Terminated *** 2024-03-16T20:47:27.067 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:26 smithi189 podman[72948]: 2024-03-16 20:47:26.797822778 +0000 UTC m=+2.916532090 container died 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T20:47:27.425 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:27 smithi189 podman[72948]: 2024-03-16 20:47:27.154297534 +0000 UTC m=+3.273006874 container cleanup 4217464e164f7ca1ee81f4c301922931c67c49e6212812f2dfb972ab837ac2de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-16T20:47:27.425 INFO:journalctl@ceph.mds.c.smithi189.stdout:Mar 16 20:47:27 smithi189 bash[72948]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-c 2024-03-16T20:47:27.578 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.c.service' 2024-03-16T20:47:27.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:27.617 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-16T20:47:27.617 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-16T20:47:27.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.f 2024-03-16T20:47:27.953 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:27 smithi189 systemd[1]: Stopping Ceph mds.f for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:28.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:27 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: 2024-03-16T20:47:27.949+0000 7f0af42ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:28.220 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:27 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f[67171]: 2024-03-16T20:47:27.949+0000 7f0af42ec700 -1 mds.f *** got signal Terminated *** 2024-03-16T20:47:31.062 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:30 smithi189 podman[73081]: 2024-03-16 20:47:30.802164142 +0000 UTC m=+2.967482610 container died 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-16T20:47:31.362 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:31 smithi189 podman[73081]: 2024-03-16 20:47:31.132723248 +0000 UTC m=+3.298041706 container cleanup 9a4b272cb36c46d61bd1b28dfea640be067200e757a4d02dbde326b55aa266ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:47:31.362 INFO:journalctl@ceph.mds.f.smithi189.stdout:Mar 16 20:47:31 smithi189 bash[73081]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-f 2024-03-16T20:47:31.564 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.f.service' 2024-03-16T20:47:31.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:31.600 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-16T20:47:31.600 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-16T20:47:31.600 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.i 2024-03-16T20:47:31.923 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:31 smithi189 systemd[1]: Stopping Ceph mds.i for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:32.219 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:31 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:47:31.918+0000 7f262e65f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:32.220 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:31 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i[67802]: 2024-03-16T20:47:31.918+0000 7f262e65f700 -1 mds.i *** got signal Terminated *** 2024-03-16T20:47:35.089 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:34 smithi189 podman[73203]: 2024-03-16 20:47:34.799582674 +0000 UTC m=+2.977934986 container died 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T20:47:35.089 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:35 smithi189 podman[73203]: 2024-03-16 20:47:35.086338662 +0000 UTC m=+3.264690973 container cleanup 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T20:47:35.470 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:35 smithi189 bash[73203]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i 2024-03-16T20:47:35.470 INFO:journalctl@ceph.mds.i.smithi189.stdout:Mar 16 20:47:35 smithi189 podman[73217]: 2024-03-16 20:47:35.279372103 +0000 UTC m=+0.478786008 container remove 73bfe9c74303bde66ca62f86d27fd948dea9a94aff0641b63fe0fbb74dfb0947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T20:47:35.503 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.i.service' 2024-03-16T20:47:35.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:35.538 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-16T20:47:35.538 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-16T20:47:35.538 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.l 2024-03-16T20:47:35.860 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:35 smithi189 systemd[1]: Stopping Ceph mds.l for 3d249ea8-e3d3-11ee-95c9-87774f69a715... 2024-03-16T20:47:36.220 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:35 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: 2024-03-16T20:47:35.855+0000 7fa5387e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T20:47:36.220 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:35 smithi189 ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l[66858]: 2024-03-16T20:47:35.855+0000 7fa5387e1700 -1 mds.l *** got signal Terminated *** 2024-03-16T20:47:39.220 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:38 smithi189 podman[73323]: 2024-03-16 20:47:38.797901978 +0000 UTC m=+3.043018455 container died 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-squid-aba3c1d, org.label-schema.schema-version=1.0) 2024-03-16T20:47:39.536 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:39 smithi189 podman[73323]: 2024-03-16 20:47:39.315497358 +0000 UTC m=+3.560613862 container cleanup 2f47d882d9006910fa921e06e67f78a704fab61e19f60da6efe32e5d79446e64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:264d2baad1e15edc24e723089ae314893f176dbcd8c8990ddeaead799c48d2b3, name=ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-squid-aba3c1d, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-16T20:47:39.536 INFO:journalctl@ceph.mds.l.smithi189.stdout:Mar 16 20:47:39 smithi189 bash[73323]: ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715-mds-l 2024-03-16T20:47:39.655 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d249ea8-e3d3-11ee-95c9-87774f69a715@mds.l.service' 2024-03-16T20:47:39.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T20:47:39.689 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-16T20:47:39.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T20:47:39.887 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:47:43.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T20:47:43.320 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:47:45.884 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T20:47:46.075 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T20:47:48.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:47:48.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:47:48.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T20:47:48.806 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T20:47:48.808 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi023/crash 2024-03-16T20:47:48.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash -- . 2024-03-16T20:47:48.854 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T20:47:48.854 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-16T20:47:48.856 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi031/crash 2024-03-16T20:47:48.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash -- . 2024-03-16T20:47:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T20:47:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-03-16T20:47:48.904 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi189/crash 2024-03-16T20:47:48.905 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash -- . 2024-03-16T20:47:48.991 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T20:47:48.991 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-03-16T20:47:48.992 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T20:47:48.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T20:47:49.083 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-16T20:27:47.079703+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T20:47:49.084 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T20:47:49.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[SEC\]' /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T20:47:49.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]' /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T20:47:49.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[WRN\]' /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-16T20:47:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-16T20:27:47.079703+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-16T20:47:49.265 INFO:tasks.cephadm:Compressing logs... 2024-03-16T20:47:49.265 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T20:47:49.308 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T20:47:49.310 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T20:47:49.348 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T20:47:49.355 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T20:47:49.355 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-16T20:47:49.355 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.10.log: -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log 2024-03-16T20:47:49.359 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T20:47:49.361 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T20:47:49.361 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-16T20:47:49.361 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T20:47:49.362 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-16T20:47:49.362 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T20:47:49.362 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-16T20:47:49.362 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-16T20:47:49.364 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-16T20:47:49.364 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.5.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T20:47:49.365 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.c.log: 92.7% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-16T20:47:49.366 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T20:47:49.366 INFO:teuthology.orchestra.run.smithi031.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T20:47:49.366 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T20:47:49.366 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-16T20:47:49.374 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-16T20:47:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T20:47:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T20:47:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T20:47:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose --/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.a.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-16T20:47:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-16T20:47:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T20:47:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T20:47:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-16T20:47:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.7% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-16T20:47:49.380 INFO:teuthology.orchestra.run.smithi023.stderr: 86.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T20:47:49.380 INFO:teuthology.orchestra.run.smithi031.stderr: 82.3% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T20:47:49.402 INFO:teuthology.orchestra.run.smithi189.stderr: 93.9% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-16T20:47:49.426 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T20:47:49.426 INFO:teuthology.orchestra.run.smithi031.stderr: 90.1% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T20:47:49.429 INFO:teuthology.orchestra.run.smithi023.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-16T20:47:49.438 INFO:teuthology.orchestra.run.smithi023.stderr: 90.0% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T20:47:49.446 INFO:teuthology.orchestra.run.smithi031.stderr: 93.7% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-16T20:47:49.502 INFO:teuthology.orchestra.run.smithi023.stderr: 82.4% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T20:47:49.509 INFO:teuthology.orchestra.run.smithi023.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T20:47:49.519 INFO:teuthology.orchestra.run.smithi189.stderr: 82.3% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T20:47:49.519 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T20:47:49.521 INFO:teuthology.orchestra.run.smithi023.stderr: 90.9% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T20:47:49.543 INFO:teuthology.orchestra.run.smithi023.stderr: 94.7% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-16T20:47:49.563 INFO:teuthology.orchestra.run.smithi031.stderr: 90.9% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T20:47:49.605 INFO:teuthology.orchestra.run.smithi023.stderr: 93.5% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-16T20:47:49.606 INFO:teuthology.orchestra.run.smithi023.stderr: 94.2% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-16T20:47:49.614 INFO:teuthology.orchestra.run.smithi189.stderr: 86.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T20:47:49.639 INFO:teuthology.orchestra.run.smithi189.stderr: 90.9% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T20:47:49.645 INFO:teuthology.orchestra.run.smithi189.stderr: 90.1% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T20:47:49.648 INFO:teuthology.orchestra.run.smithi189.stderr: 93.8% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-16T20:47:49.656 INFO:teuthology.orchestra.run.smithi031.stderr: 93.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-16T20:47:49.659 INFO:teuthology.orchestra.run.smithi031.stderr: 93.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-16T20:47:49.688 INFO:teuthology.orchestra.run.smithi031.stderr: 93.6% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-16T20:47:50.659 INFO:teuthology.orchestra.run.smithi031.stderr: 91.8% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T20:47:51.119 INFO:teuthology.orchestra.run.smithi023.stderr: 89.8% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T20:47:52.702 INFO:teuthology.orchestra.run.smithi189.stderr: 91.9% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T20:47:58.188 INFO:teuthology.orchestra.run.smithi023.stderr: 90.4% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T20:48:14.372 INFO:teuthology.orchestra.run.smithi189.stderr: 96.3% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-16T20:49:50.887 INFO:teuthology.orchestra.run.smithi189.stderr: 95.2% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-16T20:50:38.985 INFO:teuthology.orchestra.run.smithi023.stderr: 91.8% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T20:54:20.274 INFO:teuthology.orchestra.run.smithi023.stderr: 91.7% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T20:54:22.926 INFO:teuthology.orchestra.run.smithi031.stderr: 92.1% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-16T20:56:10.880 INFO:teuthology.orchestra.run.smithi031.stderr: 92.0% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T20:56:27.102 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T20:57:16.141 INFO:teuthology.orchestra.run.smithi023.stderr: 91.1% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-16T20:57:16.143 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-16T20:57:16.143 INFO:teuthology.orchestra.run.smithi023.stderr:real 9m26.822s 2024-03-16T20:57:16.143 INFO:teuthology.orchestra.run.smithi023.stderr:user 8m43.235s 2024-03-16T20:57:16.144 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m18.126s 2024-03-16T20:58:34.195 INFO:teuthology.orchestra.run.smithi031.stderr: 91.8% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T20:58:55.456 INFO:teuthology.orchestra.run.smithi189.stderr: 92.0% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-16T20:59:11.671 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T20:59:11.735 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-03-16T20:59:11.735 INFO:teuthology.orchestra.run.smithi031.stderr:real 11m22.398s 2024-03-16T20:59:11.735 INFO:teuthology.orchestra.run.smithi031.stderr:user 7m1.167s 2024-03-16T20:59:11.735 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m14.784s 2024-03-16T21:00:22.910 INFO:teuthology.orchestra.run.smithi189.stderr: 91.4% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-16T21:01:46.849 INFO:teuthology.orchestra.run.smithi189.stderr: 91.5% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T21:02:46.239 INFO:teuthology.orchestra.run.smithi189.stderr: 91.2% -- replaced with /var/log/ceph/3d249ea8-e3d3-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T21:02:46.348 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-16T21:02:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:real 14m57.022s 2024-03-16T21:02:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:user 7m20.556s 2024-03-16T21:02:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m18.309s 2024-03-16T21:02:46.348 INFO:tasks.cephadm:Archiving logs... 2024-03-16T21:02:46.349 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi023/log 2024-03-16T21:02:46.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T21:04:12.583 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi031/log 2024-03-16T21:04:12.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T21:05:21.563 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi189/log 2024-03-16T21:05:21.564 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T21:06:37.014 INFO:tasks.cephadm:Removing cluster... 2024-03-16T21:06:37.044 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force 2024-03-16T21:06:37.264 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T21:06:38.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force 2024-03-16T21:06:38.466 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T21:06:39.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d249ea8-e3d3-11ee-95c9-87774f69a715 --force 2024-03-16T21:06:39.695 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 3d249ea8-e3d3-11ee-95c9-87774f69a715 2024-03-16T21:06:40.498 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T21:06:40.498 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T21:06:40.517 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T21:06:40.536 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T21:06:40.551 INFO:tasks.cephadm:Teardown complete 2024-03-16T21:06:40.552 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-16T21:06:40.563 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-16T21:06:40.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T21:06:40.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T21:06:40.579 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-16T21:06:40.692 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T21:06:40.717 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T21:06:40.717 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T21:06:40.720 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T21:06:40.723 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T21:06:40.736 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-16T21:06:40.737 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-03-16T21:06:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-16T21:06:40.781 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T21:06:40.781 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-16T21:06:40.781 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +571us[ +571us] +/- 90ms 2024-03-16T21:06:40.781 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1750us[-1713us] +/- 71ms 2024-03-16T21:06:40.782 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +1593us[+1593us] +/- 85ms 2024-03-16T21:06:40.782 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:^- hv01.front.sepia.ceph.com 2 6 377 32 +815us[ +815us] +/- 90ms 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -1486us[-1506us] +/- 71ms 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +1840us[+1840us] +/- 85ms 2024-03-16T21:06:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T21:06:40.843 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T21:06:40.843 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-03-16T21:06:40.843 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +571us[ +571us] +/- 89ms 2024-03-16T21:06:40.843 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -1754us[-1721us] +/- 71ms 2024-03-16T21:06:40.844 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +1595us[+1595us] +/- 85ms 2024-03-16T21:06:40.844 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T21:06:40.845 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T21:06:40.859 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T21:06:40.860 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T21:06:40.897 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T21:06:41.086 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T21:06:41.196 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T21:06:41.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T21:06:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:41.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T21:06:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:41.437 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-16T21:06:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:41.474 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T21:06:41.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T21:06:41.513 INFO:teuthology.task.internal:Duration was 2912.908983 seconds 2024-03-16T21:06:41.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T21:06:41.573 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T21:06:41.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T21:06:41.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T21:06:41.581 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T21:06:41.641 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T21:06:41.661 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T21:06:41.661 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T21:06:42.159 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T21:06:42.159 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-16T21:06:42.160 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T21:06:42.182 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-03-16T21:06:42.183 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T21:06:42.215 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-16T21:06:42.216 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T21:06:42.236 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T21:06:42.236 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T21:06:42.240 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T21:06:42.261 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T21:06:42.313 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T21:06:42.313 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T21:06:42.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T21:06:42.328 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T21:06:42.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T21:06:43.079 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T21:06:43.079 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T21:06:43.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T21:06:43.147 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T21:06:43.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T21:06:43.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.314 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.339 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-16T21:06:43.360 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-03-16T21:06:43.363 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-16T21:06:43.392 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:43.418 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:43.436 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T21:06:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T21:06:43.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T21:06:43.464 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T21:06:43.466 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi023 2024-03-16T21:06:43.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T21:06:43.511 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi031 2024-03-16T21:06:43.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T21:06:43.556 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148/remote/smithi189 2024-03-16T21:06:43.556 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T21:06:43.593 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T21:06:43.593 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T21:06:43.596 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T21:06:43.598 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T21:06:43.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T21:06:43.655 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T21:06:43.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T21:06:43.682 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T21:06:43.682 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T21:06:43.685 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T21:06:43.687 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T21:06:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 21:06 /home/ubuntu/cephtest 2024-03-16T21:06:43.700 INFO:teuthology.orchestra.run.smithi031.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 21:06 /home/ubuntu/cephtest 2024-03-16T21:06:43.701 INFO:teuthology.orchestra.run.smithi189.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 21:06 /home/ubuntu/cephtest 2024-03-16T21:06:43.705 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T21:06:43.723 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T21:06:43.808 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T21:06:43.834 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-16T21:06:43.835 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T21:06:43.859 DEBUG:teuthology.nuke:shortname: smithi031 2024-03-16T21:06:43.859 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T21:06:43.864 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.086758', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhYT+mWvk67WkaGjYH/wdHRHq9iI0Lby+JQlD1OW15yt6mlEqOEsmJ7JtVqbcRCV2Ki0ncnWm6QHsmJgaCfT3o='} 2024-03-16T21:06:43.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.083363', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9MPcoe3E/e07wdBbvHbRKpaqOB46smt9Kkhd1sk0urdTVn27eReBzD3rZBqcQDu8Le07vvygTCbP/01dXPN1Q='} 2024-03-16T21:06:43.889 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-16T21:06:43.889 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T21:06:43.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-15_11:44:52-fs-wip-lusov-qdb-squid-distro-default-smithi/7602148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 20:07:14.085199', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE5c01L4cWx83yFfc9ux633eti6d9hT9EtU1qPXoT6baIaeUPmiZeHDgBUwfuoR2FFDgLvaBgtfCtm3xDvGUMOI='} 2024-03-16T21:06:43.922 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-16T21:06:43.923 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T21:06:43.945 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-16T21:06:43.945 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:43.959 INFO:teuthology.orchestra.console.smithi031:Power off 2024-03-16T21:06:43.960 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T21:06:43.973 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-16T21:06:43.984 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-03-16T21:06:43.984 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:44.000 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-16T21:06:44.000 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T21:06:44.009 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-16T21:06:44.022 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-16T21:06:44.022 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:44.048 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-16T21:06:47.974 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:48.010 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:48.050 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:48.085 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-16T21:06:48.119 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-03-16T21:06:48.159 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-16T21:06:52.090 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:52.120 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:52.161 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:52.201 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-16T21:06:52.229 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-03-16T21:06:52.271 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-16T21:06:52.302 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-16T21:06:52.330 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-03-16T21:06:56.275 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T21:06:56.384 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-16T21:06:56.485 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-16T21:06:56.590 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} duration: 2912.9089834690094 failure_reason: '"2024-03-16T20:27:47.079703+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_leonidus@teuthology success: false 2024-03-16T21:06:56.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T21:06:56.689 INFO:teuthology.run:FAIL